On Air Netlines – Linux Hint

Read Syscall Linux

On Air Netlines — Wed, 09 Jan 2019 16:05:00 +0000

So you need to read binary data? You may want to read from a FIFO or socket? You see, you may use the C standard library function, but by doing so, you will not benefit from special features provided by Linux Kernel and POSIX. For example, you may want to use timeouts to read at a certain time without resorting to polling. As well, you may need to read something without caring if it’s a special file or socket or anything else. Your only task is to read some binary contents and get it in your application. That’s where the read syscall shines.

Read a normal file with a Linux syscall

The best way to start working with this function is by reading a normal file. This is the simplest way to use that syscall, and for a reason: it doesn’t have as much constraints as other types of stream or pipe. If you think about it that’s logic, when you read the output of another application, you need to have some output ready before reading it and so you will need wait for this application to write this output.

First, a key difference with the standard library: There is no buffering at all. Each time you call the read function, you will call the Linux Kernel, and so this is going to take time –‌ it’s almost instant if you call it once, but can slow you down if you call it thousands of times in a second. By comparison the standard library will buffer the input for you. So whenever you call read, you should read more than a few bytes, but rather a big buffer like few kilobytes – except if what you need is really few bytes, for example if you check if a file exists and isn’t empty.

This however has a benefit: each time you call read, you are sure you get the updated data, if any other application modifies currently the file. This is especially useful for special files such as those in /proc or /sys.

Time to show you with a real example. This C program checks if the file is PNG or not. To do so, it reads the file specified in the path you provide in command line argument, and it checks if the first 8 bytes corresponds to a PNG header.

Here’s the code:

#include
#include
#include
#include
#include
#include
#include

typedef enum {
IS_PNG,
TOO_SHORT,
INVALID_HEADER
} pngStatus_t;

unsigned int isSyscallSuccessful(const ssize_t readStatus) {
return readStatus >= 0;

}

/*
* checkPngHeader is checking if the pngFileHeader array corresponds to a PNG
* file header.
*
* Currently it only checks the first 8 bytes of the array. If the array is less
* than 8 bytes, TOO_SHORT is returned.
*
* pngFileHeaderLength must cintain the kength of tye array. Any invalid value
* may lead to undefined behavior, such as application crashing.
*
* Returns IS_PNG if it corresponds to a PNG file header. If there's at least
* 8 bytes in the array but it isn't a PNG header, INVALID_HEADER is returned.
*
*/
pngStatus_t checkPngHeader(const unsigned char* const pngFileHeader,
size_t pngFileHeaderLength) { const unsigned char expectedPngHeader[8] =
{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A};
int i = 0;

if (pngFileHeaderLength < sizeof(expectedPngHeader)) {
return TOO_SHORT;

}

for (i = 0; i < sizeof(expectedPngHeader); i++) {
if (pngFileHeader[i] != expectedPngHeader[i]) {
return INVALID_HEADER;

}
}

/* If it reaches here, all first 8 bytes conforms to a PNG header. */
return IS_PNG;
}

int main(int argumentLength, char *argumentList[]) {
char *pngFileName = NULL;
unsigned char pngFileHeader[8] = {0};

ssize_t readStatus = 0;
/* Linux uses a number to identify a open file. */
int pngFile = 0;
pngStatus_t pngCheckResult;

if (argumentLength != 2) {
fputs("You must call this program using isPng {your filename}.\n", stderr);
return EXIT_FAILURE;

}

pngFileName = argumentList[1];
pngFile = open(pngFileName, O_RDONLY);

if (pngFile == -1) {
perror("Opening the provided file failed");
return EXIT_FAILURE;

}

/* Read few bytes to identify if the file is PNG. */
readStatus = read(pngFile, pngFileHeader, sizeof(pngFileHeader));

if (isSyscallSuccessful(readStatus)) {
/* Check if the file is a PNG since it got the data. */
pngCheckResult = checkPngHeader(pngFileHeader, readStatus);

if (pngCheckResult == TOO_SHORT) {
printf("The file %s isn't a PNG file: it's too short.\n", pngFileName);

} else if (pngCheckResult == IS_PNG) {
printf("The file %s is a PNG file!\n", pngFileName);

} else {
printf("The file %s is not in PNG format.\n", pngFileName);

}

} else {
perror("Reading the file failed");
return EXIT_FAILURE;

}

/* Close the file... */
if (close(pngFile) == -1) {
perror("Closing the provided file failed");
return EXIT_FAILURE;

}

pngFile = 0;

return EXIT_SUCCESS;

}

See, it’s a full blown, working and compilable example. Don’t hesitate to compile it yourself and test it, it really works. You should call the program from a terminal like this:

./isPng {your filename}

Now, let’s focus on the read call itself:

pngFile = open(pngFileName, O_RDONLY);

if (pngFile == -1) {
perror("Opening the provided file failed");
return EXIT_FAILURE;

}

/* Read few bytes to identify if the file is PNG. */
readStatus = read(pngFile, pngFileHeader, sizeof(pngFileHeader));

The read signature is the following (extracted from Linux man-pages):

ssize_t read(int fd, void *buf, size_t count);

First, the fd argument represents the file descriptor. I have explained a bit this concept in my fork article. A file descriptor is a int representing an open file, socket, pipe, FIFO, device, well it’s a lot of things where data can be read or written, generally in a stream-like way. I’ll go more in depth about that in a future article.

open function is one of the way to tell to Linux: I want to do things with the file at that path, please find it where it is and give me access to it. It will give you back this int called file descriptor and now, if you want to do anything with this file, use that number. Don’t forget to call close when you’re done with the file, as in the example.

So you need to provide this special number to read. Then there’s the buf argument. You should here provide a pointer to the array where read will store your data. Finally, count is how many bytes it will read at most.

The return value is of ssize_t type. Weird type, isn’t it? It means “signed size_t”, basically it’s a long int. It returns the number of bytes it successfully reads, or -1 if there’s a problem. You can find the exact cause of the problem in the errno global variable created by Linux, defined in . But to print an error message, using perror is better as it prints errno on your behalf.

In normal files – and only in this case – read will return less than count only if you have reached the file’s end. The buf array you provide must be big enough to fit at least count bytes, or your program may crash or create a security bug.

Now, read is not only useful for normal files and if you want to feel its super-powers – Yes I know it’s not in any Marvel’s comics but it has true powers – you will want to use it with other streams such as pipes or sockets. Let’s take a look on that:

Linux special files and read system call

The fact read works with a variety of files such as pipes, sockets, FIFOs or special devices such as a disk or serial port is what makes it really more powerful. With some adaptations, you can do really interesting things. Firstly, this means you can literally write functions working on a file and use it with a pipe instead. That’s interesting to pass data without ever hitting disk, ensuring best performance.

However this triggers special rules as well. Let’s take the example of a reading a line from terminal compared to a normal file. When you call read on a normal file, it only needs few milliseconds to Linux to get the amount of data you request.

But when it comes to terminal, that’s another story: let’s say you ask for an username. The user is typing in terminal her/his username and press Enter. Now you follow my advice above and you call read with a big buffer such as 256 bytes.

If read worked like it did with files, it would wait for the user to type 256 characters before returning! Your user would wait forever, and then sadly kill your application. It’s certainly not what you want, and you would have a big problem.

Okay, you could read one byte at a time but this workaround is terribly inefficient, as I told you above. It must work better than that.

But Linux developers thought read differently to avoid this problem:

When you read normal files, it tries as much as possible to read count bytes and it will actively get bytes from disk if that’s needed.
For all other file types, it will return as soon as there’s some data available and at most count bytes:
1. For terminals, it’s generally when the user presses Enter key.
2. For TCP sockets, it’s as soon as your computer receives something, doesn’t matter the amount of bytes it gets.
3. For FIFO or pipes, it’s generally the same amount as what the other application wrote, but the Linux kernel can deliver less at a time if that’s more convenient.

So you can safely call with your 2 KiB buffer without staying locked up forever. Note it can also get interrupted if the application receives a signal. As reading from all these sources can take seconds or even hours – until the other side decides to write, after all – being interrupted by signals allows to stop staying blocked for too long.

This also has a drawback though: when you want to exactly read 2 KiB with these special files, you’ll need to check read’s return value and call read multiple times. read will rarely fill your whole buffer. If your application uses signals, you’ll also need to check if read failed with -1 because it was interrupted by a signal, using errno.

Let me show you how it can be interesting to use this special property of read:

#define _POSIX_C_SOURCE 1 /* sigaction is not available without this #define. */
#include
#include
#include
#include
#include
#include
/*
* isSignal tells if read syscall has been interrupted by a signal.
*
* Returns TRUE if the read syscall has been interrupted by a signal.
*
* Global variables: it reads errno defined in errno.h
*/
unsigned int isSignal(const ssize_t readStatus) {
return (readStatus == -1 && errno == EINTR);
}
unsigned int isSyscallSuccessful(const ssize_t readStatus) {
return readStatus >= 0;
}
/*
* shouldRestartRead tells when the read syscall has been interrupted by a
* signal event or not, and given this "error" reason is transient, we can
* safely restart the read call.
*
* Currently, it only checks if read has been interrupted by a signal, but it
* could be improved to check if the target number of bytes was read and if it's
* not the case, return TRUE to read again.
*
*/
unsigned int shouldRestartRead(const ssize_t readStatus) {
return isSignal(readStatus);

}

/*
* We need an empty handler as the read syscall will be interrupted only if the
* signal is handled.
*/
void emptyHandler(int ignored) {
return;

}

int main() {
/* Is in seconds. */
const int alarmInterval = 5;
const struct sigaction emptySigaction = {emptyHandler};
char lineBuf[256] = {0};
ssize_t readStatus = 0;
unsigned int waitTime = 0;

/* Do not modify sigaction except if you exactly know what you're doing. */
sigaction(SIGALRM, &emptySigaction, NULL);
alarm(alarmInterval);

fputs("Your text:\n", stderr);

do {
/* Don't forget the '\0' */
readStatus = read(STDIN_FILENO, lineBuf, sizeof(lineBuf) - 1);

if (isSignal(readStatus)) {
waitTime += alarmInterval;
alarm(alarmInterval);

fprintf(stderr, "%u secs of inactivity...\n", waitTime);

}

} while (shouldRestartRead(readStatus));

if (isSyscallSuccessful(readStatus)) {
/* Terminate the string to avoid a bug when providing it to fprintf. */
lineBuf[readStatus] = '\0';
fprintf(stderr, "You typed %lu chars. Here's your string:\n%s\n", strlen(lineBuf),
lineBuf);

} else {
perror("Reading from stdin failed");
return EXIT_FAILURE;

}

return EXIT_SUCCESS;

}

Once again, this is a full C application that you can compile and actually run.

It does the following: it reads a line from standard input. However, every 5 seconds, it prints a line telling the user that no input was given yet.

Example if I wait 23 seconds before typing “Penguin”:

$ alarm_read
Your text:
5 secs of inactivity...
10 secs of inactivity...
15 secs of inactivity...
20 secs of inactivity...
Penguin
You typed 8 chars. Here's your string:
Penguin

That’s incredibly useful. It can be used to update often the UI to print the progress of the read or of the processing your application you’re doing. It can also be used as a timeout mechanism. You could also get interrupted by any other signal that might be useful for your application. Anyway, this means your application can now be responsive instead of staying stuck forever.

So the benefits outweighs the drawback described above. If you wonder whether you should support special files in an application normally working with normal files – and so calling read in a loop – I would say do it except if you’re in a hurry, my personal experience often proved that replacing a file with a pipe or FIFO can literally make an application much more useful with small efforts. There’s even premade C functions on Internet that implements that loop for you: it’s called readn functions.

Conclusion

As you can see, fread and read might look similar, they’re not. And with only few changes on how read works for the C developer, read is much more interesting for designing new solutions to the problems you meet during application development.

Next time, I will tell you how write syscall works, as reading is cool, but being able to do both is much better. In the meantime, experiment with read, get to know it and I wish you an Happy New Year!

malloc in c language

On Air Netlines — Thu, 27 Dec 2018 02:41:12 +0000

You can come here for two reasons: either you want to dynamically allocate content, or you want to know more about how malloc works. In either case, you’re at the right place! Dynamic allocation is a process that happens a lot but generally we don’t use it ourselves: the vast majority of programming languages are managing memory for you as it’s an hard job and if you fail to do it properly, there’s security implications.

However, if you’re doing C, C++ or assembly code, or if you implement a new external module in your favorite programming language, you will need to manage your dynamic memory allocation yourself.

What’s dynamic allocation? Why I need malloc?

Well, in all applications, when you create a new variable – it’s often called declaring a variable – you need memory to store it. As your computer is in the modern days, it can run more than one application at a time and so, each application should tell to your OS (here Linux) that it needs that amount of memory. When you write this kind of code:

#include
#include
#define DISK_SPACE_ARRAY_LENGTH 7

void getFreeDiskSpace(int statsList[], size_t listLength) {
return;
}

int main() {
/* Contains the free disk space of the last 7 days. */
int freeDiskSpace[DISK_SPACE_ARRAY_LENGTH] = {0};
getFreeDiskSpace(freeDiskSpace, DISK_SPACE_ARRAY_LENGTH);

return EXIT_SUCCESS;
}

The freeDiskSpace array needs memory so you’ll need to ask Linux for approval to get some memory. However, as it’s obvious when reading the source code that you’ll need an array of 7 int, the compiler automatically asks Linux for it, and it’ll allocate it on the stack. This basically means that this storage gets destroyed when you return the function where the variable is declared. That’s why you can’t do that:

#include
#include

#define DISK_SPACE_ARRAY_LENGTH 7

int* getFreeDiskSpace() {
int statsList[DISK_SPACE_ARRAY_LENGTH] = {0};

/* WHY ARE WE DOING THAT?! statsList will be DESTROYED! */
return statsList;
}

int main() {
/* Contains the free disk space of the last 7 days. */
int *freeDiskSpace = NULL;
freeDiskSpace = getFreeDiskSpace();

return EXIT_SUCCESS;
}

You see more easily the problem now? Then, you want to concatenate two strings. In Python and JavaScript, you would do:

newStr = str1 + str2

But as you know, in C it doesn’t work like this. So to build an URL for example, you need to concatenate two strings, such as URL path and domain name. In C, we’ve strcat, right, but it only works if you’ve an array with enough room for it.

You’ll be tempted to know the length of the new string by using strlen, and you would be right. But then, how would you ask Linux to reserve this unknown amount of memory? Compiler can’t help you: the exact space you want to allocate is only known at runtime. That’s exactly where you need dynamic allocation, and malloc.

Writing my first C function using malloc

Before writing code, a little explanation: malloc allows you to allocate a specific number of bytes for your application usage. It’s really simple to use: you call malloc with the number of bytes you need, and it returns a pointer to your new area that Linux reserved for you.

You have only 3 responsibilities:

Check if malloc returns NULL. That happens when Linux has not enough memory to provide.
Free your variables once unused. Otherwise you’ll waste memory and it’ll slow down your application.
Never use the memory zone after you have freed the variable.

If you follow all these rules, all will go well and dynamic allocation will solve you many problems. Because you choose when you free the memory, you can also safely return a variable allocated with malloc. Just, don’t forget to free it!

If you wonder how to free a variable, it’s with the free function. Call it with the same pointer than malloc returned you, and the memory is freed.

Let me show you with the concat example:

#include
#include
#include

/*
* When calling this function, don’t forget to check if the return value is NULL
* If it’s not NULL, you must call free on the returned pointer once the value
* is no longer used.
*/
char* getUrl(const char* const baseUrl, const char* const toolPath) {

size_t finalUrlLen = 0;
char* finalUrl = NULL;

/* Safety check. */
if (baseUrl == NULL || toolPath == NULL) {
return NULL;
}

finalUrlLen = strlen(baseUrl) + strlen(toolPath);

/* Don’t forget the ’\0’, hence the + 1. */
finalUrl = malloc(sizeof(char) * (finalUrlLen + 1));

/* Following malloc rules... */
if (finalUrl == NULL) {
return NULL;
}

strcpy(finalUrl, baseUrl);
strcat(finalUrl, toolPath);
return finalUrl;
}

int main() {

char* googleImages = NULL;
googleImages = getUrl("https://www.google.com", "/imghp");

if (googleImages == NULL) {
return EXIT_FAILURE;
}

puts("Tool URL:");
puts(googleImages);

/* It’s no longer needed, free it. */
free(googleImages);
googleImages = NULL;

return EXIT_SUCCESS;
}

So you see a practical example for using dynamic allocations. First, I avoid pitfalls such as giving getUrl return value straight to puts function. Then, I also take the time to comment and document the fact the return value should be freed properly. I also check for NULL values everywhere so anything unexpected can be safely caught instead of crashing the application.

Finally, I take the extra care of freeing the variable and then setting the pointer to NULL. That avoids to be tempted to use – even by mistake – the now freed memory zone. But as you can see, it’s easy to free a variable.

You may notice that I used sizeof in malloc. It allows to know how many bytes a char is using and clarifies the intent in the code so it’s more readable. For char, sizeof(char) is always equal to 1, but if you use an array of int instead, it works exactly the same way. For example, if you need to reserve 45 int, just do:

fileSizeList = malloc(sizeof(int) * 45);

This way, you quickly see how much you want to allocate, that’s why I always recommend its usage.

How works malloc under-the-hood?

malloc and free are, in fact, functions included in all C programs that will talk to Linux on your behalf. It will also makes dynamic allocation easier because, at start, Linux doesn’t allow you to allocate variables of all sizes.

Linux provides two ways to get more memory in fact: sbrk and mmap. Both have limitations, and one of them is: you can allocate only relatively big amounts, such as 4,096 bytes or 8,192 bytes. You can’t request 50 bytes like I did in the example, but you also can’t request 5,894 bytes.

This has an explanation: Linux needs to keep a table where it tells which application has reserved which memory zone. And this table uses space as well, so if every byte needed a new row in this table, a big share of memory would be needed. That’s why memory is splitted in big blocks of, for example, 4,096 bytes, and much like you can’t buy 2 oranges and a half in a grocery, you can’t ask for half blocks.

So malloc will take these big blocks and gives you a little slice of these memory blocks whenever you call it. As well, if you freed few variables, but not enough to justify freeing a whole block, malloc system may keep blocks and recycle memory zones when you call malloc again. This has the benefit to make malloc faster, however memory reserved by malloc can’t be used in any other application, while the program isn’t currently using it in reality.

But malloc is smart: if you call malloc to allocate 16 MiB or a big amount, malloc will probably ask Linux for full blocks dedicated just for this big variable by using mmap. This way, when you call free, it will more likely avoid that waste of space. Don’t worry, malloc is doing a way better job at recycling than humans do with our garbage!

Conclusion

I think now you better understand how all of that works. Of course, dynamic allocation is a big topic and I think we can write a full book on the topic, but this article should make you comfortable with the concept both in general and with practical programming advices.

Next Generation Cron With systemd: Creating A Timer

On Air Netlines — Tue, 18 Dec 2018 11:34:22 +0000

Do you need to schedule some task in the future on your computer? This may look simple – after all, your dishwasher is able to wait before launching with the help of a button – but sometimes computers is making such simple tasks so hard.But if you have some background, you will have probably heard of cron, this piece of software fully dedicated to launch the right task at the right time. But this tool has really been designed with simplicity in mind and you may in the end have bad surprises.If you ever succeeded to schedule a task on Windows, you have used the Windows Task Planner. It does have a GUI by default but it doesn’t make it so simple to use as well: these two systems just launch a process at a fixed time and date.

In order to understand how systemd can be helpful to you there, I’ll take an example.

What pitfalls systemd timers will avoid you?

If you ever own a machine with data you care about in it, you’ll want to have a copy of your data in another, probably safer place. If you manage a server, it’s mandatory: after all, how will you recover if your hard disk fails and prevent you to recover any data?

So as a responsible person you set up backup every week or every day. You may set it up using cron, you schedule it at 4:24 AM, but here starts the problem: what if your server is shutdown from 4:10 AM to 4:30 AM for any reason?

Well it’s likely cron will just skip that backup. This might be critical if that happens often and silently or if your code relies on the fact it runs and it may fail otherwise. Generally this happens when you set up a cleanup task via cron and it doesn’t launch. Suddenly your code may have insufficient space to continue and will break – it’s sad, so sad situation, right Mr Elton John.

However, if a missed launch can be a problem, imagine one second – wow, John Lennon now? – that your task is too much slow. If your task is set to run every 10 minutes but takes 15 minutes to complete, cron or Windows will happily launch another task even if the current task isn’t gone yet – and so, you’ll have 2 instances of your task running concurrently, which is the perfect recipe for disaster. When a program is running concurrently while it’s not designed to do so, it will really likely corrupt files, other softwares, databases – and your server becomes suddenly a sinking ship like Titanic.

OK, maybe I am going too far with Titanic but you get the idea. While systemd couldn’t have done much to save this ship, it can help you with all these shortfalls and ensure you a longer Christmas vacation thanks to the bugs it will avoid you. It’s time now to get to know how to set up systemd timers.

How to schedule automated server backup?

First of all, systemd timers triggers a systemd service, so before scheduling your task, you’ll need to make it first a service. Fortunately, I’ve written a guide to create systemd service, this way it will introduce you with systemd’s way of working. You should read it before going on. Unless if you exactly know what you’re doing, your systemd service file should not contain any WantedBy= setting. If you want to start your service at a specific time, you probably don’t want to start it on boot.

Thanks to the systemd service system, it’s impossible to have multiple instances of your task running by mistake: if a task is already running, it will just skip that launch and leave the currently running task finish its job.

Once you have a systemd service to schedule, create a file with the same filename as your service except that it should end with .timer instead of .service. In our automated backup example, the service would be automated-backup.service and the timer would be automated-backup.timer. Both files should be in the same directory. As I told you in the systemd service article, I recommend you to write these files in a normal place such as your home directory and then, copy them to a systemd folder, once you’ve finished your edits.

So, let me show you what our timer file is looking like:

[Unit]
Description=Schedule backups during off peak hours

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=7200

Persistent=true

[Install]
WantedBy=timers.target

Much like in systemd services, there’s 3 sections. [Unit] or [Install] work exactly the same as explained in my systemd services article. Please note that WantedBy= is important here because timers can be started or stopped, so if you don’t tell systemd to start your timer during boot, it will never trigger. timers.target is a special systemd target for timers.

Now, the [Timer] section. Inside it, you’ll find all settings related to when the timer should trigger. For our automated backup, I’ve told systemd to run it between 3 AM and 5 AM at server’s timezone. The exact time is random on each day.

OnCalendar= sets the timer related to your server’s time (wallclock), such as every Sundays at 1 PM. If you have used cron previously, you should be really familiar with this syntax. However it has some added benefits.

For example, if you want something to happen hourly, you can do like this:

OnCalendar=hourly

and daily:

OnCalendar=daily

In fact, it supports all of the following values:

minutely
hourly
daily
monthly
weekly
yearly
quarterly
semiannually

There’s however a problem with these keywords: for example, daily triggers always a midnight, which is often a peak hour in computing systems. That’s why it’s recommended to use RandomizedDelaySec= (its usage is specified below). Anyway for backup it’s not a good option: midnight isn’t off peak hours, it’s rather the reverse. So we need to set more accurately when we want to see that task launched.

If you want more control, you can write a date like 2018-12-06 12:49:37. Well if you’re that specific you’ll just trigger the timer once. To make it recurrent, you’ll replace any of these elements by * asterisk.

OnCalendar=*-*-* 03:00:00

As you can see above, in our backup example, all the date part is *-*-*, meaning it should occur every day of every month of every year. Now if you do:

OnCalendar=*-12-25 03:00:00

Then it runs every 25th December at 3 AM. Perfect systemd timer for Santa Claus – even if I doubt he will ever need one! So asterisk adds recurrence where you put it. If you put it in year field, it means “every year”, etc.

Finally, you can add UTC at the end of the line to use UTC time instead of local timezone. For example, some services reset their API quotas at midnight but to avoid any timezone bias it uses UTC. So for such tasks, you would do:

OnCalendar=daily UTC

Now, let’s solve another problem: rush hours. systemd has also a setting to fight against that.

RandomizedDelaySec= allows to delay the task of a random amount of time. The value is the max number of seconds the timer will delay. It’s specifically intended for such cases. You remember that in systemd, daily always triggers at midnight? Well, weekly always triggers at Monday midnight, and yearly triggers at January 1 midnight, one of the worst peaks in the year with network outages everywhere. You certainly don’t want that to happen.

By adding a delay, you remove that problem: it will automatically delay at a unknown time your task. Randomness here is important because it’s far more likely to be even when it’s random and an even load allows to better optimize your tasks.

Say you need to run your tasks around 7 AM for the morning but you want to allow a small delay of max 15 minutes, you would do like this:

RandomizedDelaySec=900

That should be enough for delays. Sometimes even milliseconds delays are enough to prevent unintended spikes.

Persistent= takes care of missed timer triggers. What if your server is shutdown during night? Well, the backup would never trigger at all. Setting it to true allows systemd run it on the next boot in such cases. This way you know in one way or another, the timer’s task will be run. Its usage is simple, you just do this:

Persistent=true

This has however one drawback that’s really hard to avoid anyway: when multiple tasks from different timers are missed, they will all run at boot and slow down that boot. In my opinion that’s much better than if it never runs and after all that’s normal, the most appropriate moment to run the timer is when it is scheduled, afterwards it will be probably inappropriate anyway.

OnBootSec= is the last option I’ll show you (but not the least). It’s if you want to trigger a timer some time after boot instead of based on calendar. For example, if you need to check on startup if your server is started properly and working as intended, you could write a check service and use that timer setting to trigger it after the system had enough time to boot.

Let’s say the system needs 3 minutes to boot, you could do:

OnBootSec=180

And despite its name, you can also do:

OnBootSec=3 minutes

If you precise both OnBootSec= and OnCalendar=, it will start the service whenever any of these 2 events happen.

Okay, now it’s time to save your file, copy it to the system folder if you followed my advice above, and test if your timer is working properly.

Enable your new timer and monitoring

In order to test your new timer, you have to tell systemd you added a new timer, so you need to type this command:

$ sudo systemctl daemon-reload

Now, systemd will take in account your new timer and look closely on when to run your task. As systemd is always running, it’s after all one of the best candidates to manage and run your scheduled tasks.

One thing you might find counterintuitive though: a timer is by default disabled. In order to enable it, you need to do this command:

$ sudo systemctl enable --now automated-backup.timer

You will then probably want to see if your timer acts as expected. Good news: systemd is even kind enough to have a command telling you when it was last launched and when the next launch is scheduled (except if timer is set to run only at boot, as systemd doesn’t know when the system will boot again, obviously). Here’s that command:

$ systemctl status automated-backup.timer

Finally, when you no longer need the timer, you can disable it as well:

$ sudo systemctl disable --now automated-backup.timer

Conclusion

Using systemd timers, your management of scheduled tasks are to a next level: honestly, I personally feel like scheduled tasks should have been this way since years.

Oh, one little surprise for you: all systemd timers are logged in a well structured system with filtering, log rotation and all the like. So I invite you to see how you can see logs about your scheduled tasks!

systemd unit file creating a service

On Air Netlines — Sun, 11 Nov 2018 04:13:43 +0000

Service management is something you don’t even think of when you use your Linux workstation or Linux server everyday, but when it’s not there you will really hate it. When you create for example a new server program that needs to run 24/7, doing this challenge without service management is a nightmare where you create in fact a small service system yourself, which will be obviously not as good as the manager developed by a full team during years, anyway.

With its services, systemd makes all of this easier, really easier. As soon as you want something monitoring your application and easy control of it, systemd is the way to go, and that’s what I’m going to explain here!

Where are Systemd Services

To add a new service, well, you need to answer this question. As always in systemd, it depends if the service is only for your user or the whole system. We’ll focus on how systemd works for whole system services.

The exact location depends of why and how the service got installed. If the service is installed by a package manager, it will be generally in /usr/lib/systemd/system. For software you develop or the ones that doesn’t support systemd by itself, you will put the service file in /usr/local/lib/systemd/system. Please keep in mind though that some distributions doesn’t support this folder in /usr/local. Finally, if you want to configure an existing systemd service, /etc/systemd/system is the way to go.

Inside these folders you can find multiple file extension such as *.socket, *.target or *.service. Obviously we’re going to focus on the last. systemd uses the filename as the name of the service when starting it or stopping it etc. So generally filenames in service only contains alphanumeric characters along with hyphens and underscores. During development I recommend to create it in your documents and then copy it to systemd location when done, that would avoid you problems if you save in middle of editing.

OK so please create your service file in your documents. Now we’re ready to review how to write this file.
[Note: See potential bug report in comment section of this blog post]

[Unit]
Description=Penguins Web Application HTTP server (running in port 8080)
WantedBy=multi-user.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /usr/local/bin/penguin-web-app/main.py
Restart=always

The file format is in fact close to ini. I know it may be weird given ini files are often found in Windows but that’s how it works. The service file is first divided in 2 sections: [Unit] and [Service]. Each section configures a specific aspect of systemd: [Unit] contains elements shared by all systemd unit files while [Service] is only for configuration specific to setting up a new service.

Then the section is configured with properties such as Description= or ExecStart=. The value is separated from property name by the equal sign = without any space.

Let’s go back to the file shown above. It describes a service designed to run a web app written in Python about penguins. systemd will restart it whenever the process exits and starts the server upon server’s start-up if you enable it with systemctl enable command. Cool eh?

But you’re maybe your next web app isn’t about penguins — and that’s a shame — and it’s not written in Python. In this case you’ll want to learn more about the possible configurations.

Properties of Systemd Services

Let’s first focus about the properties in [Unit]:

Description= is just about giving a clear description of what service is doing. It’s displayed in service list, service logs so you want it to be descriptive but it should stay in one line and one sentence.

WantedBy= allows to say to systemd: when this thing is started, starts me as well. Generally you’ll put the name of a target. Examples of common targets:

multi-user.target: when server is OK and is ready to run command line applications

graphical.target: when GNOME or KDE is ready

network-up.target: when server is connected properly to a network

OK for the beginning these properties of [Unit] is enough. Let’s take a look on [Service] now.

Type= helps systemd in how to know if a service is running. Here are common types:

simple is probably the most commonly used: systemd considers the process you launch as the one doing the service. If the process stops, it considers the service stopped as well, etc.

forking is preferred for applications that were written to be a server but without the help of a service management system. Basically it expects the launched process to fork and that fork is considered the final process for the service. In order to be more accurate, you may also help systemd with a PID file, where the PID of the process to track is written by the launched application.

ExecStart= is probably the most important for a service: it precises what application to launch when starting the service. As you can see in the Penguin service, I have used /usr/bin/python3 and not python3 straight away. It’s because systemd documentation explicitly recommends to use absolute paths in order to avoid any surprises.

But that’s also for another reason. Other services’ management system tend to be based on Shell scripts. However systemd, for performance reason, doesn’t run a shell by default. So you can’t provide directly a shell command in ExecStart=. You can however still use a shell script by doing:

ExecStart=/usr/bin/bash /usr/local/bin/launch-penguin-server.sh

Not that hard right? Note that if you need to run some process to signal your service to stop cleanly, ExecStop= exists, as well as ExecReload= for reloading services.

Restart= allows you to explicitly tell when the service should be restarted. This is one of the important features of systemd: it ensures that your service stays up as long as you wish to, so pay close attention to this option.

Restart= Meaning

always systemd will keep restarting it whenever it terminates or crashes. Well, until you do systemctl stop service-name.service.

It’s perfect for servers and online services as you prefer few useless restarts over having to manually restart the service without any reason.

on-abnormal When the service process crashes, restart the service. However, if the application exits cleanly, don’t restart it.

It’s more useful for cron-jobs like services that needs to do a task reliably but don’t need to run all the time.

on-failure Much like on-abnormal, but it also restarts the service when the application exits cleanly but with a non-zero exit code. Non-zero exit codes generally means an error happened.

no systemd will not restart the service automatically.

Generally useful to get access to other systemd features such as logging without the restart feature.

WorkingDirectory= can enforce a working directory when launching your application. The value must be an absolute directory path. Working directory is used when you use relative paths in your application’s code. For our penguins service, it could be:

WorkingDirectory=/srv/penguin-web-app/

Then, security is important so you generally want to not launch your service with root privileges. User= and Group= enables you to set the user or group name or UID/GID under which your application will be launched. For example:

User=penguin-web
Group=penguin-web

EnvironmentFile= is a powerful option. Applications running as services often needs configuration and environment files allows to set that configuration in two ways:

The application can read directly the environment variable.

But also you can set different command line arguments to your application without changing the service file.

The syntax of this file is simple: you type the environment variable name, the equal sign = and then its value. Then you put the absolute path of your environment file into EnvironmentFile property.

So example:

EnvironmentFile=/etc/penguin-web-app/environment

And the /etc/penguin-web-app/environment file contains:

LISTEN_PORT=8080

Then our penguins web app will have access to LISTEN_PORT environment variable and listen to the expected port.

Save and Start the Newly Created Systemd Service

So if you followed my advice, you edited your service file in your home directory. Once you’re satisfied, copy that file to /usr/local/lib/systemd/system, assuming your distribution supports that path. The filename of your service file will be its service name. This filename have to end with .service. For example, for our penguins server, it would be penguin-web-app.service.

Then, you have to tell systemd you added a new service, so you need to type this command:

$ sudo systemctl daemon-reload

Okay now systemd is aware of your new service, assuming your file doesn’t contain a syntax error. After all, it’s your first file so it’s likely you’ll make mistakes. You have to run this command above on every update in your service file.

Now, time to start the service:

$ sudo systemctl start penguin-web-app.service

If it fails with a Unit not found error such as this one:

$ sudo systemctl start penguin-web-app.service
Failed to start penguin-web-app.service: Unit not found.

It means that your distribution doesn’t support the directory or you didn’t named correctly your service file. Be sure to check out.

If you set up your service with WantedBy= and wants that your service starts automatically, you have to enable it, with this command:

$ sudo systemctl enable penguin-web-app.service

The cool thing with a service is that it runs in background. The problem: how to know if it runs properly and if it’s running if it’s running in background? Don’t worry, systemd team thought about that too and provided a command to see if it runs properly, since how much time, etc:

$ systemctl status penguin-web-app.service

Conclusion

Congrats! You can now have your applications managed without you caring about restarting it manually every time. Now, I recommend you to read our other article about systemd logs: Master journalctl: understand systemd logs. With that you can use the powerful logging system on your new service and build more reliable servers!

FFMPEG Images To Video Tutorial

On Air Netlines — Mon, 24 Sep 2018 11:27:56 +0000

So you’re interested in building a video based on images? Linux softwares can help you with that task too, and especially one: ffmpeg.

If you don’t know it already, ffmpeg is a software that does video and audio conversion and editing using filters. It’s the one which supports the greatest number of different codecs. If you already used VLC, then you know a bit ffmpeg: VLC uses ffmpeg to decode as many videos as possible.

But before making videos, I need to tell you some concepts so you don’t get lost in the way.

Video concepts: how images can work in a video

It’s not really hard to integrates images in a video. And there’s a reason: videos are based on a series of images. Let me explain.

A video (even a YouTube video) is a suite of still images that changes quickly. In cinema and movie theaters, between each image there’s a black picture because the mechanism has to switch frame and was too slow. But it’s not noticeable by human eye because the mechanism is fast enough and because of an optical illusion.

But computers don’t have this problem. LCD screens just keep displaying the last image. Anyway, you can in fact extract a independent image from every image of a video. When an image comes from a video, it’s called a frame. That’s exactly what you see when you pause a video – and when you do so faces generally doesn’t look good!

Generally, videos from United States a contains 30 frames per second – realize a bit. It’s 1,800 frames per minute, 54,000 frames per half hour of video, or 108,000 frames per hour. That’s a lot and that’s how you wonder sometimes how a single image can weight 1 MiB but one minute of 1080p video can weight only 15 MiB.

The number of pictures per second is called the frame rate. At 30 frames per second, you change image every 33 milliseconds. So concretely, if you wanted to do one minute of video based solely on your images, you would need to have 1,800 JPG or PNG files per minute.

That’s useful if you want to do advanced video editing without using video tools: you just have to build images for each frame with your effects and then you can bundle it in a video. But sometimes you want a still image to last because, well, you want a still image in video.

Okay, now, time to install ffmpeg.

Install ffmpeg on Linux

Depending of your distribution, installing ffmpeg can be easy or a little tricky. It’s a free software with source code publicly available, but as it can decode or encode patented formats like MP4, some distributions excludes it from their repositories. For example, in Red Hat Enterprise Linux, CentOS and Fedora, you need RPMFusion because of patents. Moreover, all Debian versions and Ubuntu before 16.04 is distributing a false version of ffmpeg based on a illegitimate fork.

But before hurrying in installing it, maybe the right version already installed? Let’s test:

$ ffmpeg -version
ffmpeg version X.XXXXXXXX Copyright (c) 2000-2018 the FFmpeg developers

If after the copyright you do see “the FFmpeg developers”, you have the original version of FFMpeg, you need to do nothing. However, if you see:

$ ffmpeg -version
ffmpeg version X.XXXXXXXX Copyright (c) 2000-2018 the Libav developers

Then it means you’re using a fork of ffmpeg called Libav. Debian and older versions of Ubuntu silently substitute FFMpeg with Libav. If it tells you it’s deprecated, please ignore it, that’s misleading. If you have the fork installed it’s time to remove your version of false ffmpeg and then follow the guide below to install the right version. Probably like this:

$ sudo apt-get remove ffmpeg

So now you may haven’t it installed yet or you have the wrong version, time to install it!

In Fedora, Red Hat Enterprise Linux (RHEL) and CentOS, go to https://rpmfusion.org/Configuration and enable free RPMFusion repository on your computer. Then, if you’re in Fedora, do:

$ sudo dnf install ffmpeg

And for CentOS & Red Hat Enterprise Linux, do:

$ sudo yum install ffmpeg

That’s all for Fedora and Red Hat based systems, it’s installed.

In all versions of Debian (and all derivatives) and Ubuntu before 16.04, you need to get FFMpeg static build from the official website. Go to https://ffmpeg.org/download.html#build-linux and below “Linux Static Builds”, click on “32-bit and 64-bit for kernel 2.6.32 and above”. Then, below “release: X.X.X”, download the appropriate archive, extract it and you can launch the executables located in extracted folder via CLI.

For the ones using Ubuntu 16.04 and above, it’s easier, just do:

$ sudo apt-get install ffmpeg

Phew! Finally we should be good to go! Before going on, test a last time with:

$ ffmpeg -version

ffmpeg version X.XXXXXXXX Copyright (c) 2000-2018 the FFmpeg developers

Create videos from many images

So, for our first example, we’ll create a video from Images where each image represents a frame that will be displayed for 33 milliseconds. Let me show you first the command.

** MP4 and H.264 are patented codecs, please check if you have the right to encode with it. **

$ ffmpeg -r:v 30 -i "Penguins - %05d.png" -codec:v libx264 -preset veryslow
-pix_fmt yuv420p -crf 28 -an "Penguins.mp4"

OK so how that works? For this command to work, you need to have many frames where each frame is a file like Penguins – 00043.png. This command will then combine all frames in order at a rate of 30 FPS. So, Penguins – 00043.png comes before Penguins – 00044.png in the video and ffmpeg will respect that. So if you have 120 frames, your video will have a 4 seconds duration.

There’s few requirements about format, width and height for frames to work. If you don’t follow these rules, ffmpeg may ignore some images or cancel video creation process. So:

All frames in a same video must share:

Width & height

Color depth

Frames should be in a standard video size such as:

640 ✕ 360 (360p)

853 ✕ 480 (480p)

1280 ✕ 720 (720p)

1920 ✕ 1080 (1080p)

4096 ✕ 2306 (4K)

Prefer frames in PNG rather than JPG

Avoid transparency or alpha in PNG format

The command will then take some time to convert the video, optimize its performance and make you a MP4 file named Penguins.mp4.

Now, you may want to have, well, a still image remaining for more than 33 milliseconds. In this case, your video is more a slideshow and it’s not the same affair. To do so, you may first give a slower framerate for the input and then tell ffmpeg to duplicate frames in output. No, YouTube and Vimeo won’t really appreciate a 0.5 FPS video even if you change your image every 2 seconds.

Let’s rather do that:

$ ffmpeg -r:v 1/5 -i "Penguins - %05d.png" -r:v 30 -codec:v libx264 -preset veryslow
-pix_fmt yuv420p -crf 28 -an "Penguins.mp4"

There we go! ffmpeg will make each of your images appear for 5 seconds but in a 30 FPS video. Don’t worry about the duplicate frames warning: it’s what you want.

Conclusion

Now, you’re – the new Spielberg – able to create your own videos. You can do a simple slideshow or prepare a video frame by frame using an image editor such as GIMP, but you can also benefit from the large set of effects and codecs you can find in ffmpeg.

Take the time to experiment by yourself – you’re an artist, after all – and create a video that will do millions of views in YouTube!

Your First C Program Using Fork System Call

On Air Netlines — Tue, 28 Aug 2018 11:52:35 +0000

By default, C programs have no concurrency or parallelism, only one task happens at a time, each line of code is read sequentially. But sometimes, you have to read a file or – even worst – a socket connected to a remote computer and this takes really a long time for a computer. It takes generally less than a second but remember that a single CPU core can execute 1 or 2 billions of instructions during that time.

So, as a good developer, you will be tempted to instruct your C program to do something more useful while waiting. That’s where concurrency programming is here for your rescue – and makes your computer unhappy because it has to work more.

Here, I’ll show you the Linux fork system call, one of the safest way to do concurrent programming.

Concurrent programming can be unsafe?

Yes, it can. For example, there’s also another way calling multithreading. It has the benefit to be lighter but it can really go wrong if you use it incorrectly. If your program, by mistake, reads a variable and write to the same variable at the same time, your program will become incoherent and it’s almost undetectable – one of the worst developer’s nightmare.

As you will see below, fork copies the memory so it’s not possible to have such problems with variables. Also, fork makes an independent process for each concurrent task. Due to these security measures, it’s approximately 5x slower to launch a new concurrent task using fork than with multithreading. As you can see, that’s not much for the benefits it brings.

Now, enough of explanations, it’s time to test your first C program using fork call.

The Linux fork example

Here is the code:

#include
#include
#include

#include
#include

int main() {
pid_t forkStatus;

forkStatus = fork();

/* Child... */
if        (forkStatus == 0) {
printf("Child is running, processing.\n");
sleep(5);
printf("Child is done, exiting.\n");

/* Parent... */
} else if (forkStatus != -1) {
printf("Parent is waiting...\n");

wait(NULL);
printf("Parent is exiting...\n");

} else {
perror("Error while calling the fork function");

}

return 0;

}

I invite you to test, compile and execute the code above but if you want to see what the output would look like and you are too “lazy” to compile it – after all, you are maybe a tired developer who compiled C programs all day long – you can find the output of the C program below along with the command I used to compile it:

$ gcc -std=c89 -Wpedantic -Wall forkSleep.c -o forkSleep -O2
$ ./forkSleep
Parent is waiting...
Child is running, processing.
Child is done, exiting.
Parent is exiting...

Please don’t be afraid if the output isn’t 100% identical to my output above. Remember that running things at the same time means that tasks are running out-of-order, there’s no predefined ordering. In this example, you might see that child is running before parent is waiting, and there’s nothing wrong with that. In general, the ordering depends of the kernel version, the number of CPU cores, the programs that are currently running on your computer, etc.

OK, now get back to the code. Before the line with fork(), this C program is perfectly normal: 1 line is executing at a time, there’s only one process for this program (if there was a small delay before fork, you could confirm that in your task manager).

After the fork(), there’s now 2 processes that can run in parallel. First, there’s a child process. This process is the one that has been created upon fork(). This child process is special: it hasn’t executed any of the lines of code above the line with fork(). Instead of looking for the main function, it will rather run the fork() line.

What about the variables declared before fork?

Well, Linux fork() is interesting because it smartly answers this question. Variables and, in fact, all the memory in C programs is copied into the child process.

Let me define what is doing fork in a few words: it creates a clone of the process calling it. The 2 processes are almost identical: all variables will contain the same values and both processes will execute the line just after fork(). However, after the cloning process, they are separated. If you update a variable in one process, the other process won’t have its variable updated. It’s really a clone, a copy, the processes shares almost nothing. It’s really useful: you can prepare a lot of data and then fork() and use that data in all clones.

The separation starts when fork() returns a value. The original process (it’s called the parent process) will get the process ID of the cloned process. On the other side, the cloned process (this one is called the child process) will get the 0 number. Now, you should start to understand why I’ve put if/else if statements after the fork() line. Using return value, you can instruct the child to do something different of what the parent is doing – and believe me, it’s useful.

On one side, in the example code above, the child is doing a task that takes 5 seconds and prints a message. To imitate a process that takes a long time, I use the sleep function. Then, the child exits successfully.

On the other side, the parent prints a message, wait until child exits and finally prints another message. The fact parent wait for its child is important. As it’s an example, the parent is pending most of this time to wait for its child. But, I could have instructed the parent to do any kind of long-running tasks before telling it to wait. This way, it would have done useful tasks instead of waiting – after all, this is why we use fork(), no?

However, as I said above, it’s really important that parent waits for its childs. And it’s important because of zombie processes.

How waiting is important

Parents generally want to know if childs have finished their processing. For example, you want to run tasks in parallel but you certainly don’t want the parent to exit before childs are done, because if it happened, shell would give back a prompt while childs have not finished yet – which is weird.

The wait function allows to wait until one of the child processes is terminated. If a parent calls 10 times fork(), It will also need to call 10 times wait(), once for each child created.

But what happens if parent calls wait function while all childs have already exited? That’s where zombie processes are needed.

When a child exits before parent calls wait(), Linux kernel will let the child exit but it will keep a ticket telling the child has exited. Then, when the parent calls wait(), it will find the ticket, delete that ticket and the wait() function will return immediately because it knows the parent needs to know when the child has finished. This ticket is called a zombie process.

That’s why it’s important that parent calls wait(): if it doesn’t do so, zombie processes remains in memory and Linux kernel can’t keep many zombie processes in memory. Once limit is reached, your computer is unable to create any new process and so you will be in a very bad shape: even for killing a process, you may need to create a new process for that. For example, if you want to open your task manager to kill a process, you can’t, because your task manager will need a new process. Even worst, you cannot kill a zombie process.

That’s why calling wait is important: it allows the kernel clean up the child process instead of keep piling up with a list of terminated processes. And what if the parent exits without ever calling wait()?

Fortunately, as the parent is terminated, no one else can call wait() for these childs, so there’s no reason to keep these zombie processes. Therefore, when a parent exits, all remaining zombie processes linked to this parent are removed. Zombie processes are really only useful to allow parent processes to find that a child terminated before parent called wait().

Now, you may prefer to know some safety measures to allow you the best usage of fork without any problem.

Simple rules to have fork working as intended

First, if you know multithreading, please don’t fork a program using threads. In fact, avoid in general to mix multiple concurrency technologies. fork assumes to work in normal C programs, it only intends to clone one parallel task, not more.

Second, avoid to open or fopen files before fork(). Files is one of the only thing shared and not cloned between parent and child. If you read 16 bytes in parent, it will move the read cursor forward of 16 bytes both in the parent and in the child. Worst, if child and parent write bytes to the same file at the same time, the bytes of parent can be mixed with bytes of the child!

To be clear, outside of STDIN, STDOUT, & STDERR, you really don’t want to share any open files with clones.

Third, be careful about sockets. Sockets are also shared between parent and childs. It’s useful in order to listen a port and then let have multiple child workers ready to handle a new client connection. However, if you use it wrongly, you will get in trouble.

Fourth, if you want to call fork() within a loop, do this with extreme care. Let’s take this code:

/* DO NOT COMPILE THIS */
const int targetFork = 4;
pid_t forkResult

for (int i = 0; i < targetFork; i++) {
forkResult = fork();
/* ... */

}

If you read the code, you might expect it to create 4 childs. But it will rather create 16 childs. It’s because childs will also execute the loop and so childs will, in turn, call fork(). When the loop is infinite, it’s called a fork bomb and is one of the ways to slow down a Linux system so much that it no longer works and will need a reboot. In a nutshell, keep in mind that Clone Wars isn’t only dangerous in Star Wars!

Now you have seen how a simple loop can go wrong, how to use loops with fork()? If you need a loop, always check fork’s return value:

const int targetFork = 4;
pid_t forkResult;
int i = 0;

do {
forkResult = fork();
/* ... */

i++;

} while ((forkResult != 0 && forkResult != -1) && (i < targetFork));

Conclusion

Now it’s time for you to do your own experiments with fork()! Try novel ways to optimize time by doing tasks across multiple CPU cores or do some background processing while you wait for reading a file!

Don’t hesitate to read the manual pages via the man command. You will learn about how fork() precisely works, what errors you can get, etc. And enjoy concurrency!

Master journalctl: understand systemd logs

On Air Netlines — Fri, 03 Aug 2018 14:26:01 +0000

Systemd is the new tool managing services. Created initially by Red Hat, it allows to better manage services via a centralized process that monitors and launches services as needed. But systemd also includes a container system, a cron system, a way to provide temporary directories to services in a secure way and also a logging system – that’s where we’re going to focus here.

Understanding logs is important: if you ever fall on a server that has a bug or is hacked, generally your only way to understand what happened is via logs. The main application we’re going to use is journalctl hence the name of the article. So listen carefully as on the right day, you may be happy to know how it works.

Where are stored systemd logs? And what format it is stored in?

We’ll take assumption you have a normal system, because systemd can be customized to be in exceptional places. As well, some Linux distributions like Ubuntu 16.04 disabled persistent logging by default, which prevent systemd to do its job correctly. If you have such distribution, edit /etc/systemd/journald.conf file, change Storage=auto to Storage=persistent and finally, reboot.

So you will find normally the systemd logs files in /var/log/journal. The journalling system is itself a service called system-journald.service. Let’s try to list the files in this directory:

# ls /var/log/journal/ -R
/var/log/journal/:
15e43c1734090ac7fbea6b40fcd99d31

/var/log/journal/15e43c1734090ac7fbea6b40fcd99d31:
system@a39da368947bd2ba-231f9bfc18a7a356.journal~
system@62ac1299826d036cb043d6c06a9493b7-0000000000000001-00067d6410099a19.journal
user-1000@b27e98812223a9bc-387e0521703f73d9.journal~
user-1000@2123bc076b58569fe1fb13e9dbc1b0e0-0000000000000001-0007fe36ac2810e0.journal
user-1000.journal
[lots of other files like the ones above...]

Because I want you to keep reading, I had to shorten the output as it contains many files (in my example, more than 60 files), sorry about that! Tempted to open one maybe?

# head --bytes=512 /var/log/journal/15e43c1734090ac7fbea6b40fcd99d31/user-1000@2123bc076
b58569fe1fb13e9dbc1b0e0-0000000000000001-0007fe36ac2810e0.journal
?s,q?n/FLz???Ulz?l?]????
?_?b???z????o?y1KN ?i?eO??W?u? ?=?x0?L?d?7??X4n#?e? d3l?
p??o|MFO:?!qs?.tK??R?\??1?|5 ????$?g??#?S??;??B7???????t???Y????mN?q????ZQ
?Yv?e?????BD?C?? wF??d|
?2?? 7???????[??Un?=8????c?2=p?&?"   ?0
????*????_?? ???
5?????yk?G? ?6?|??u??w: #12?Y??
3      TU;???'?jX??2?x`?=??[Q???@Z
T_???3eHMd?@?_?>??3S???,lR?.?$?g?L???s?/E??M1??q???

Hey, see, that doesn’t really looks like the usual log files you see right? Don’t worry, this file isn’t corrupted, you just discovered an aspect of systemd: systemd stores files in a binary format. That’s why it’s as small as possible: structured data such as time or location is stored straight in binary, which generally takes less bytes than does text. But that’s not the only reason.

systemd doesn’t only store log lines. Its intent is to make logs monitoring and exploration easier. To help in this task, log messages are in fact a line of text accompanied with data such as log severity (warning, error, etc.), or even fields that would be only useful to your application (URL requested for example).

# journalctl --output=verbose --all
PRIORITY=6
_UID=0
_GID=0
_CAP_EFFECTIVE=3fffffffff
_BOOT_ID=ee4cc2ce7e8273aaffb5fc59c873ce7b
_MACHINE_ID=bc422e0feaab64bb7dd218c24e6830e5
_HOSTNAME=linux
SYSLOG_FACILITY=3
SYSLOG_IDENTIFIER=systemd
UNIT=dnf-makecache.service
_TRANSPORT=journal
_PID=1
_COMM=systemd
_EXE=/usr/lib/systemd/systemd
_CMDLINE=/usr/lib/systemd/systemd --switched-root --system --deserialize 76
_SYSTEMD_CGROUP=/init.scope
_SYSTEMD_UNIT=init.scope
_SYSTEMD_SLICE=-.slice
_SELINUX_CONTEXT=system_u:system_r:init_t:s0
CODE_FILE=src/core/job.c
CODE_LINE=795
CODE_FUNCTION=job_log_status_message
MESSAGE_ID=a76e08846f5f0971371dbb11126e62e1
MESSAGE=Started dnf makecache.
# journalctl --catalog --lines=3000 --pager-end "_TRANSPORT=kernel"    RESULT=done
_SOURCE_REALTIME_TIMESTAMP=1532886335471422

I have told you there’s a lot of fields (here there’s 25 fields, or 29 counting timestamps), all of the snippet above is only for a single log message! The big benefit is that you can run a search by filtering on any field in this log message. This really enables you to advanced filtering.

One of the most obvious filter you would want is to filter by the service. As you can see above, there’s a UNIT field so you can easily filter to get only log messages from one service. I’ll tell you more about that later.

But this amount of data also means something else: in nearly all cases, you’ll never open a log file manually and you won’t ever touch to the /var/log/journal folder. You’ll use journalctl for any task related to logging. There’s no such log rotation thing, everything is managed by log message time.

As well, the number of fields will depend on how good is the integration of systemd in your application. The more fields a log message contains, the better it is. For base system services, systemd already took care of doing a good integration but for other applications and services, the quality of integration varies a lot. Normally, this should get better over time as people gets used to systemd.

Okay, now it’s time to discover journalctl’s features.

Most used commands for journalctl

The first command you might want to take a look on is the one showing the Linux kernel’s logs. Yes, systemd also handles the storage of kernel’s logs, so you can get the logs of previous boots as well. Here is the command:

# journalctl --catalog --lines=3000 --pager-end "_TRANSPORT=kernel"

It shows you a pager where you can see the last messages. You can scroll up to the last 3,000 lines using arrow keys (↑ / ↓) or Page Up / Page Down. The –catalog flag instructs journalctl to show context around log lines, much like computer reboots or, in other contexts, a service stopping / starting. I always put this flag as context always matters, it helps to know in which situation the log line appeared, so you can guess why you got this log line.

Now, maybe you want to only see the log lines from the current boot:

# journalctl --catalog --lines=35000 --pager-end --boot "_TRANSPORT=kernel"

Note the –boot command line argument works in all situations, not only with kernel’s logs. If you prefer to start from the beginning:

# journalctl --catalog --boot "_TRANSPORT=kernel"

I don’t know if it’s the case for you, but I’ve enough of kernel logs! And what about having a general overview of your machine?

# journalctl --catalog --lines=3000 --pager-end

Wow, there’s a lot of things happening on your system! A bit of filtering would be helpful here. One of the most used filters is matching a specific service (like your SSH server or HTTP server), the systemd unit filename for SSH service is sshd.service, so:

# journalctl --catalog --lines=3000 --pager-end --unit=sshd.service

That’s cool, isn’t it? Well it’s only usable if you know the name of the service – but in lot of cases, you don’t know the name of that service. If you’re on such situation, you may want a listing of the services, their descriptions and their status:

# systemctl list-units --type=service

Okay, this problem is now solved. But sometimes, you have an error message you get from an external system like your own website or from an application on your desktop. So you will probably want to search a specific word or sentence in the log message. Since systemd v237, it’s now possible.

In journalctl, the search is case insensitive if the word you search is all in lowercase. So if you search the word port, it will also search the word port with capitalized letters. An example:

# journalctl --catalog --lines=3000 --pager-end --grep="port"

Now, if you search a word like CPU, it will only search CPU with all capitalized letters, it won’t search cpu.

# journalctl --catalog --lines=3000 --pager-end --grep="CPU"

You remember the error message from the external system? Generally, these messages contains a timestamp. To filter the log message, you may want to use that timestamp. journalctl can list you all log messages since a specific date and time with the –since argument:

# journalctl --catalog --since="2018-07-30 09:30:00"

If that external system is remote or is using UTC timestamps, you’ll want to filter based on a UTC date and time and display in the terminal the UTC timestamps so you don’t need to convert it in your head, that tends to be really confusing. To do so, you’ll need to add UTC after the time string in –since argument. You will then need to add the –utc flag. So, for example:

# journalctl --catalog --since="2018-07-30 10:45:00 UTC" --utc

Note you can use the –utc flag alone, in this case it will basically display all dates and times in UTC timezone.

# journalctl --catalog --lines=3000 --pager-end --utc

Logs are better managed with journalctl

As you can see with all previous commands, systemd journaling makes filtering and so debugging easier as you can select through all log lines using a single command, journalctl. Some of you probably knew ancient times where you had to manually open every file in /var/log to have a general idea of the problem and of what’s happened. With all the tips you learned here, you’ll own solid tools to look at your log messages in the way YOU want it.

Install Fedora on Google Compute Engine

On Air Netlines — Sun, 03 Dec 2017 06:32:21 +0000

How to enjoy Fedora in Google Compute Engine

So, you too you’ve been disappointed to see there’s no prebuilt image of Fedora from Google in Google Compute Engine (GCE)? The good news is that, thanks to this missing image, you’ll build your own custom image and so learn an important aspect of Google Cloud Platform (GCP). This means extensive customization of your VMs if you want it.

Before to start, a brief thing you need to know. VMs are really much like computers, but you already know that, right? What you may not know is that, images, in GCE, are prebuilt OS that the virtual computer will have on its first start up. It’s much like when you buy a computer, you get it with (sadly) a preinstalled version of Windows installed on the hard disk. And when you boot up the first time, it’ll boot this preinstalled version that is the same for all computers of this model/manufacturer.

In Google Compute Engine, it’s all the same. When you create an instance, you need to start somewhere, so it will let pick you a preinstalled Linux to boot from, also called an “image”. Note that some VM users will say “In VMs, usually, we start booting via an ISO CD with a setup assistant”, but usually Google Compute Engine VMs are intended to run unattended, and a setup GUI would basically prevent that.

So in this article, we’re going to:

Borrow the latest official Fedora Cloud image.

Add some software on top of it so it is better compatible with Google Compute Engine.

Package it as a GCP image.

Create an instance using this image.

This all in Google Compute Engine.

Get Fedora Cloud image for customization

To start, you need to create a VM where we’re going to build and modify the official Fedora Cloud image. So, create an instance with the following options:

Give it a name, choose the right zone, etc.
Please keep the zone in mind because we’ll need it later.

In “Machine Type”, choose the “f1-micro”. This is more than enough for our needs.

In “Boot Disk”, click “Change” and choose “CentOS 7”. This is the closest image to Fedora (Fedora is maintained by Red Hat, CentOS is RHEL without customer support) and using familiar tools will help to build the image.

In “Identity and API access”, choose “Allow all access to Cloud APIs”. This is for simplicity, as we’ll need to use gcloud a lot and creating a service account is more cumbersome.
As it’s only a VM that’ll last few minutes, that’s not a problem. Don’t use that in production setup with automated builds of images, though.

You might want to make the VM “Preemptible”, as Preemptible VMs costs much less. Note though that if you do, Google can shutdown your VM at anytime and you’ll have to restart the VM and resume where you left off.

Click on the “Create” button. The most fun moment of cloud administration is this one, if you ask me.

Give it 2 minutes to start and then, SSH into the VM using the “SSH” button. It will open a window with SSH connected to your brand new CentOS 7 VM.

The first thing you need is to install wget. You can install curl if you prefer, but the article will use wget.

$ sudo yum install wget

Then, once installed, go to https://alt.fedoraproject.org/cloud/ and next to “Cloud Base compressed raw image”, right click on “Download” and copy the address link.

Get back to the VM, and do the following:

$ wget "{PASTE URL HERE}"

This will download the file. Fedora Servers, their mirrors and Google has a great infrastructure, so the download is going to last only few seconds. Probably my second favorite moment of cloud administration!

Once done, run this command:

$ xz --decompress --keep "Fedora-Cloud-Base-XX-X.X.x86_64.raw.xz"

Note that you have to adapt the filename depending of the version you download. This is going to extract a sparse file of ~3 GB that we can then loop-mount for second step. It is going to take one minute, so take a coffee break and come back when done.

Preparing Fedora for the Google Cloud Platform’s ride

OK, so what do we call preparation here? Roughly, it is loop mounting the raw disk, chroot inside it, add some software so it can use all GCP features and then finally clean up various temporary files.

OK, let’s mount it:

$ mkdir boot $ sudo mount -o loop,offset=1048576 "$PWD/Fedora-Cloud-Base-XX-X.X.x86_64.raw" "$PWD/boot"

Once again, adapt the file name.

Okay, I see that you don’t really understand this command line, so time for an explanation. This command says to Linux: Take a file from disk, act as if it was a disk partition and try to mount it. This is the principle of the loop mount. But you’ll also notice the “offset=1048576”. There’s an offset because this raw disk is a disk, not a partition. It comes partitioned, with a bootloader on it, so the VM knows what to do on startup. But we can’t mount, or chroot into a bootloader, right?

So by setting the offset, Linux is in fact, mounting the first partition of the raw disk stored in the file. It is an ext4 partition and to leave enough space to bootloaders, first partitions generally starts 1 MiB after the beginning of the disk. Hence the offset. Next:

$ cd boot $ sudo mount --bind /dev dev && sudo mount --bind /sys sys && sudo mount --bind /proc proc && sudo mount --bind /etc/resolv.conf etc/resolv.conf $ sudo chroot ./ /usr/bin/bash

And now, welcome to your Fedora loop-mounted raw chroot! So, why all of that? First, we mount anything needed for any decent application to work, /dev, /proc and /sys. Also, we mount bind resolv.conf because otherwise the chroot has no Internet access (!). Finally, we chroot into it. Note that we use /usr/bin/bash because /bin in Fedora is a symlink to /usr/bin.

Now, it’s time to install the Google Cloud Platform software to make it work well.
The first thing you might want to do is have an up-to-date image. It’s better, no? So:

# dnf upgrade --assumeyes --nogpgcheck "*"

Once again an occasion to take a sip of coffee, since it is going to take a while. The “–nogpgcheck” is because GPG check and chroot doesn’t act very well to each other. Then, do this:

# cat > "/etc/yum.repos.d/google-cloud.repo" <<"EOR" [google-cloud-compute] name=Google Cloud Compute baseurl=https://packages.cloud.google.com/yum/repos/google-cloud-compute-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOR

And do:

# dnf install --nogpgcheck --assumeyes google-compute-engine python-google-compute-engine

This is going to install all Google-related software in order to be best compatible with Google Compute Engine. For example, it will allow you to check/uncheck IP forwarding from Google Cloud Platform interface, or use SSH in browser instead of having to explicitly create a SSH key for the VM. Next:

# touch "/.autorelabel" # dnf clean all

As you know, one of the best things about Fedora, it is its security features and enterprise-level quality, and SELinux is a part of it. So in order to avoid headaches, it retriggers a relabel of the whole disk upon VM’s first start up.

It does so because the labels in SELinux are wrong in a chroot environment and forgetting this little step makes the VM unbootable and unreachable from the outside. The dnf upgrade above rewrites a lot of the core files that are unlabeled and then SELinux prevents these binaries from running. Note that it means the first VM startup can take a few minutes before it is ready.

dnf clean up allows to keep the image as small as possible. This saves you the cost of repeatedly storing things you don’t need.

Time to exit chroot:

# exit $ cd ../

Now you got out of the loop-mounted directory, you can unmount bind-mounted things:

$ sudo umount boot/dev boot/proc boot/sys boot/etc/resolv.conf

And then, let’s do this:

$ sudo fstrim --verbose boot

This helps you to keep the loop-mounted image even smaller. Basically, during the upgrade, the raw image will be quickly filled with zones of temporary files. Unlike real hard disks, when a file is deleted in a raw image, it is just deleted in filesystem metadata of the raw image and it still uses space on the hard disk hosting the raw image. fstrim allows you to make these unused zones “sparse” and so this space of deleted files is given back to the disk.

Unmount the loop-mounted device now:

$ sudo umount boot $ mv "Fedora-Cloud-Base-XX-X.X.x86_64.raw" "disk.raw" $ tar --create --auto-compress --file="Fedora-Cloud-Base-XX-X.X.x86_64.tar.gz" --sparse disk.raw

OK, cool, you have now your final image, pre-packaged! The size for me is around 350 MiB, tiny eh? Now, you remember when I said you had to take note of the zone? It is now you need it!

Head to Google Cloud Storage and create a bucket. I assume here you don’t have already a bucket in the right zone, otherwise it is perfectly fine to use a pre-existing one. So create a bucket with the following options:

Give it a name.

Choose “Regional” type. Since we only use the bucket here for images, which can be regenerated easily, regional allows to pay less by not having a geo-redundant backup of the file.

Pick the region where the CentOS VM you created is located.

Hit Create.

Wait for the bucket to be created, and once done, go in SSH window again and do:

$ gsutil cp "Fedora-Cloud-Base-XX-X.X.x86_64.tar.gz" "gs://[name of the bucket]/"

This copies the packaged image to Google Cloud Storage so we can say to GCP: Take that .tar.gz and make it an image.

Now, you can shutdown the instance at that point. Don’t delete it yet as we’ll test the Fedora instance before deleting this build VM.

Now in Google Compute Engine, get inside “Images”. Hit the “Create Image” button. Configure it like this:

Name it “fedora-cloud-XX-YYYYMMDD” where XX is the version and YYYYMMDD is the today’s year, month and date.

In “Family”, enter “fedora-cloud-XX”.

In “Source”, choose “Cloud Storage file”.

Click on the “Browse” button, get in your bucket, and select the .tar.gz file uploaded earlier.

Create the image.

And that’s all folks!

Testing phase

OK but that wouldn’t be a real how-to guide if we did not test whether it works as expected. So to see if it worked great, get to “VM Instances” and then click on “Create Instance”.

Configure the instance this way:

While Fedora Cloud can work on almost all VM shapes, I recommend you to choose the cheapest VM type, f1-micro, since we only use this VM for testing purposes.

Below “Boot disk”, click on the “Change” button.
Go in the “Custom Image” tab and then pick the image you just created.
Don’t forget to set the boot disk size. It will be set to below 4 GB, way too small. The minimum size of Google Cloud Platform disks is 10 GB and the recommended minimum by Google is 200 GB.

Once again, you might want to set the VM as Preemptible, especially if you’ll use it only for test purposes and not keep it.

Click on the “Create” button.

Now, you’ve to wait 5 minutes, enough time to clean up your keyboard! And after these 5 minutes, now you can click the “SSH” button.

And now, hopefully, hooray, you’re logged in to your Fedora VM, run by Google Cloud! At that point, don’t forget to delete the test VM and the build VM.

I hope you enjoyed the tutorial, and it will work nicely for you. That’s all folks (for real this time), and see you in a Fedora VM!

Restart=	Meaning
always	systemd will keep restarting it whenever it terminates or crashes. Well, until you do systemctl stop service-name.service. It’s perfect for servers and online services as you prefer few useless restarts over having to manually restart the service without any reason.
on-abnormal	When the service process crashes, restart the service. However, if the application exits cleanly, don’t restart it. It’s more useful for cron-jobs like services that needs to do a task reliably but don’t need to run all the time.
on-failure	Much like on-abnormal, but it also restarts the service when the application exits cleanly but with a non-zero exit code. Non-zero exit codes generally means an error happened.
no	systemd will not restart the service automatically. Generally useful to get access to other systemd features such as logging without the restart feature.