Thursday, May 10, 2018

Linux sandboxing improvements in Firefox 60

Continuing our past work, Firefox 60 brings further important improvements to security sandboxing on Linux, making it harder for attackers that find security bugs in the browser to escalate those into attacks against the rest of the system.

The most important change is that content processes — which render Web pages and execute JavaScript — are no longer allowed to directly connect to the Internet, or connect to most local services accessed with Unix-domain sockets (for example, PulseAudio).

This means that content processes have to follow any network access restrictions Firefox imposes — for example, if the browser has been set up to use a proxy server, connecting directly to the internet is no longer possible. But more important are the restrictions on connections to local services: they often assume that anything connecting to them has the full authority of the user running it, and either allow it to ask for arbitrary code to run, or aren't careful about preventing that. Normally that's not a security problem because the client could just run that code itself, but if it's a sandboxed Firefox process, that could have meant a sandbox escape.

In case you encounter problems that turn out to be associated with this feature, the `security.sandbox.content.level` setting described previously can be used for troubleshooting; the network/socket isolation is controlled by level 4. Obviously we'd love to hear from you on Bugzilla too.

Additionally, System V IPC is now blocked. This is a relatively old way of communicating between processes on Unix systems, which modern software mostly avoids. It's not needed in content processes, and it can grant access to resources they shouldn't have, so we now block it. (There are exceptions: ATI's legacy `fglrx` graphics drivers and VirtualGL require it, so if either of those is in use, we grant an exception. This does not affect newer AMD drivers, who can run with these restrictions in place.)

These changes also allow something else useful: because content processes no longer need any direct access to the network or filesystem (they can still ask the main process for some whitelisted subset they are allowed to read), we can ask the kernel to take it away.

Specifically, we use Linux namespaces and chroot. This is similar to how containers like Docker work, and complements the existing sandbox: in addition to blocking specific system calls like `open` and `connect`, we can prevent filesystem/network access no matter how it is requested. This is part of our defense in depth: if a clever attacker manages to bypass one layer of protection, that still won't be enough to compromise the system.

The kernel feature we use for this is called “unprivileged user namespaces”. Although it has been in Linux for a while, some distributions don't support it because they rely on older kernels, or they disable it because of security concerns. Although we don't want to take a position on the latter, obviously if the feature is available, Firefox might as well make use of it to make itself more secure. The Sandbox section of about:support will tell you whether it is supported on your distribution or not. If it isn't, you are still secure, just missing one of the extra hardening layers.

The one exception to the network policy, for now, is the X11 protocol which is used to display graphics and receive keyboard/mouse input. Content processes don't need most of its functionality, but they still have some dependencies on it. Hardening work on X11 is ongoing and will be addressed in future releases.

Developer information about Unprivileged User Namespaces:

User namespaces allow a process to behave as if it were root in a restricted subset of the system by granting it extra capabilities. Firefox uses the thus gained CAP_SYS_CHROOT capability to chroot() the content processes into an empty directory, thereby blocking their view of the real filesystem. Using user namespaces avoids the need to install a setuid root binary to achieve this.

Unprivileged user namespaces had some security bugs in their initial Linux implementation, and some Linux distribution maintainers are still wary of keeping them enabled. The problem is that although the process behaves as if it were root in its own namespace, at many levels in the kernel checks must be done to ensure it cannot do things that it couldn't do in its original namespace, where it is not root. There have been some calls to rework the feature to reduce the possibility of errors in this regard.

In this context, we'd like to remark that an application like Firefox only needs CAP_SYS_ADMIN, CAP_SYS_CHROOT, CAP_SET(UG)ID inside the namespace to achieve most effect, and specifically does not require the CAP_NET_ADMIN permission that has been proven rather bug prone in the past.

Thursday, November 9, 2017

Linux sandboxing improvements in Firefox 57

Firefox 57 not only ships a large amount of performance improvements and a UI refresh, it also contains a number of technological improvements under the hood. One of these is that the security sandbox was tightened, making it harder for attackers - should they find a security hole in Firefox in the first place - to escalate that attack against the rest of the system, or your data.

The content process - that is the one that renders the web pages from the internet and executes JavaScript - is now blocked from reading large parts of the filesystem, with some exceptions for libraries, configuration information, themes and fonts. Notably, it is no longer possible to read private information in the home directory or the Firefox user profile, even if Firefox were to be compromised.

We could not block the web content rendering entirely from reading the filesystem because Firefox still uses GTK directly - we draw webpage HTML widgets using the same theming as the native desktop. Rather than postpone the security improvements till this is reworked, we've elected to work around this by allowing a few very specific locations through. Similar things apply to the use of PulseAudio (to be fixed around Firefox 59 with a new Rust based audio backend), ffmpeg (media decoding must stay sandboxed) and WebGL.

We've made sure this works on all common, and many not-so common configurations. So most of you can stop reading here, but for those who like to know more details or are tinkerers, the following might be of interest. Due to the infinite configurability of Linux systems, it's always possible there will be cases where a non-standard setup can break things, and we've kept Firefox configurable, so you can at least help yourself, if you're so inclined.

For example, we know that in Firefox 57, allowing your system font configuration to search for fonts from the same directory as where your downloads are stored (a rather insecure configuration, for that matter!) can cause these fonts to appear blank in web-pages.

The following settings are available in about:config:


This determines the strictness of the sandbox. 0 disables everything, 1 filters dangerous system calls, 2 additionally blocks writing to the filesystem, and 3 adds blocking of (most) reading from the filesystem. This is a high level knob, use it only to quickly check if an issue is caused by sandboxing. After changing this, you'll have to restart Firefox for it to take effect.

If lowering security.sandbox.level fixes your problems, turn it back to the default value (3 in Firefox 57) and restart Firefox with the MOZ_SANDBOX_LOGGING=1 environment variable set, which will log any accesses the Sandbox allows or blocks. "Denied" messages will give you a clue what is being blocked. Don't forget to file a bug in Bugzilla, so we can track the problem and if possible, make things work by default.


List of paths (directories and files) that Firefox is additionally allowed to read from, separated by commas. You can add things here if Firefox can't reach some libraries, config files or fonts that are in a non-standard location, but avoid pointing it to directories that contain personal information.


List of paths that Firefox is additionally allowed to write to, separated by commas. It should almost never be necessary to change this.


List of system call numbers that Firefox will additionally allow, separated by commas. A disallowed system call will crash Firefox with a message mentioning "seccomp violation". It should almost never be necessary to change this. We'd particularly like to hear from your in Bugzilla if you require this.

Friday, October 7, 2016

Firefox sandbox on Linux tightened

As just announced on, we landed a set of changes in today's Nightly that tightens our sandboxing on Linux. The content process, which is the part of Firefox that renders webpages and executes any JavaScript on them, had been previously restricted in the amount of system calls that it could access. As of today, it no longer has write access to the filesystem, barring an exception for shared memory and /tmp. We plan to also remove the latter, eventually.

As promised, we're continuing to batten down the hatches gradually, making it harder for an attacker to successfully exploit Firefox. The changes that landed this night are an important step, but far from the end of the road, and we'll be continuing to put out iterative improvements.

Some of our next steps will be to address the interactions with the X11 windowing system, as well as implementing read restrictions.

Friday, May 20, 2016

Technical Debt, Episode 1

One of the projects I'm working on for Mozilla is our Content Sandboxing. We've been using sandboxing for a while to protect some plugins like Flash, as well as media plugins, but now that Firefox can render webpages in a separate process, we can apply restrictions to what those "Web Content" processes can do, too. Those processes are the part of Firefox that is essentially exposed to the internet, and hence to potentially dangerous webpages.

Although we go to great lengths to make this impossible, there is always a chance that a bug in Firefox would allow an attacker to exploit and take over a Web Content process. But by using features provided by the operating system, we can prevent them from taking over the rest of the computing device by disallowing many ways to interact with it, for example by stopping them from starting new programs or reading or writing specific files.

This feature has been enabled on Firefox Nightly builds for a while, at least on Windows and Mac OS X. Due to the diversity of the ecosystem, it's taken a bit longer for Linux, but we are now ready to flip that switch too.

The initial version on Linux will block very, very little. It's our goal to get Firefox working and shipping with this first and foremost, while we iterate rapidly and hammer down the hatches as we go, shipping a gradual stream of improvements to our users.

One of the first things to hammer down is filesystem access. If an attacker is free to write to any file on the filesystem, he can quickly take over the system. Similarly, if he can read any file, it's easy to leak out confidential information to an attacking webpage. We're currently figuring out the list of files and locations the Web Content process needs to access (e.g. system font directories) and which ones it definitely shouldn't (your passwords database).

And that's where this story about technical debt really starts.

While tracing filesystem access, we noticed at some point that the Web Content process accesses /etc/passwd. Although on most modern Unix systems this file doesn't actually contain any (hashed) passwords, it still typically contains the complete real name of the users on the system, so it's definitely not something that we'd want to leave accessible to an attacker.

My first thought was that something was trying to enumerate valid users on the system, because that would've been a good reason to try to read /etc/passwd.

Tracing the system call to its origin revealed another caller, though. libfreebl, a part of NSS (Network Security Services) was reading it during its initialization. Specifically, we traced it to this array in the source. Reading on what it is used for is, eh, quite eyebrow-raising in the modern security age.

The NSS random number generator seeds itself by attempting to read /dev/urandom (good), ignoring whether that fails or not (not so good), and then continuing by reading and hashing the password file into the random number generator as additional entropy. The same code then goes on to read in several temporary directories (and I do mean directories, not the files inside them) and perform the same procedure.

Should all of this have failed, it will make a last ditch effort to fork/exec "netstat -ni" and hash the output of that. Note that the usage of fork here is especially "amusing" from the sandboxing perspective, as it's the one thing you'll absolutely never want to allow.

Now, almost none of this has ever been a *good* idea, but in its defense NSS is old and caters to many exotic and ancient configurations. The discussion about /dev/urandom reliability was raised in 2002, and I'd wager the relevant Linux code has seen a few changes since. I'm sure that 15 years ago, this might've been a defensible decision to make. Apparently one could even argue that some unnamed Oracle product running on Windows 2000 was a defensible use case to keep this code in 2009.

Nevertheless, it's technical debt. Debt that hurt on the release of Firefox 3.5, when it caused Firefox startup to take over 2 minutes on some people's systems.

It's not that people didn't notice this idea was problematic:
I'm fully tired of this particular trail of tears. There's no good reason to waste users' time at startup pretending to scrape entropy off the filesystem.
-- Brendan Eich, July 2009
RNG_SystemInfoForRNG - which tries to make entropy appear out of the air.
-- Ryan Sleevi, April 2014
Though sandboxing was clearly not considered much of a use case in 2006:
Only a subset of particularly messed-up applications suffer from the use of fork.
-- Well meaning contributor, September 2006
Nevertheless, I'm - still - looking at this code in the year of our Lord 2016 and wondering if it shouldn't all just be replaced by a single getrandom() call.

If your system doesn't have getrandom(), well maybe there's a solution for that too.

Don't agree? Can we then at least agree that if your /dev/urandom isn't secure, it's your problem, not ours?

Friday, April 4, 2014

Only what unites us

If anything, the last week has given us quite some things where we can have great discussions about. I'd love to do a treatise on how free speech interacts with the ability to do Internet witch burnings, or how universal truths relate to political opinions, but the nerves are tense and the knives sharpened, so attempting to do so would most likely only be preaching to the converted. Maybe another time.

Yet, when looking at the questions for the upcoming Mozillians Town Hall meeting, there is a pattern that worries me, and I would like to address.

Mozilla is a community that organizes around a mission. The mission is set out in the Manifesto. The Manifesto uses rather broad language, e.g. "must enrich the lives of individual human beings".

If we have to learn anything from the past 10 days, it is that we can only survive as a community if we interpret this mission only in its most narrow scope, where we can and should find common ground. Attempting to read the Manifesto in the widest possible manner and presuming to find that all of our fellow Mozillians have done so in the same way is the road to failure as a group and a community. Our cultural differences are immense and things which we find self-evident can be unimaginable to other. We should group among the narrow set of goals that unites us, not among what divides us.

Mission creep is death. As a result, Mozilla is pragmatic. We've thrown efforts under the bus when we believed it to be necessary to survive. It personally still pains me that we threw VP8 under the bus in favor of H264, but choices have to be made about what hurts us least.

Now, if anyone wants to make the argument that an "open, participatory and accessible" Internet obviously has a direct and inalienable relation to an already-repealed law in the state of California in the United States of America that tried to refine the legal definition of marriage, so Mozilla must fight for this cause, then fine. I am not going to argue with you. I simply want to point out that the remainder of this post is not addressed to you.

Why do I want to make this point now? I have looked at the list of questions for the Town Hall Meeting, and I could summarize a fair number of the highest up voted ones as "Why do we think it is acceptable that the personal opinion of a Mozillian influences his career prospects in our community?"

I understand this question. I've struggled with it myself. Up until yesterday, I wouldn't have believed Brendan would step down simply because of the enormous implication that has in relation to the above. When I joined Mozilla, I was encouraged by my manager (Stuart or Doug, sorry forgot which of you two!): "We expect our contributors to have an opinion and speak out on it". This obviously does not mix with what has happened this week.

The answer though, is above: it is not our fight to fight. Should a political/moral opinion and monetary support for it be grounds for a week-long internet shitstorm ending in resignation? The Fox has no opinion on this issue. The fact that we have such wildly differing opinions on it is a clear sign: this is far from the Mozilla mission, and it's outside our scope until we can find common ground. Should we fight global warming? The Fox has no opinion. Should abortion be allowed? The Fox most certainly has no opinion on that one. Should we legalize marihuana? The Fox wonders what you've been smoking that you're even asking him. Should we be able to anonymously participate on the Internet? You bet.

The "why do our private opinions affect our work" question is inappropriate for the Town Hall, because it is not Mozilla's problem. We have to choose to focus only on what unites us.

Wednesday, April 17, 2013

WebRTC support on Android

Yesterday, WebRTC support landed in Firefox for Android, which means it is available in today's Nightly. Wait a minute, you might say, didn't you demo this already at MWC a month and a half ago? Well yes. But to get the code used for that demo landed in the main Firefox repository (instead of a branch), some cleanup work was needed, and we needed to make sure that older, pre-ICS phones didn't just blow up when you tried to make a phone call on them.

Although, in theory, WebRTC should work on any device Firefox for Android supports, for an ideal experience you're going to want to have an Ice Cream Sandwich or Jelly Bean device. We'll do what we can for older devices, but realistically Android was missing some essential APIs before that time (especially before Android 2.3) and device performance will also limit what experience you get. Nevertheless I had no problems getting a video call up on my Galaxy Tab 10.1 with Android 3.2, and many devices will probably work just fine.

By default WebRTC is included in Firefox for Android, but behind a preference that is off by default. To enable, flip these preferences to "true":
  • media.navigator.enabled
  • media.peerconnection.enabled

Secondly, because the UI is currently very provisional, you might want to disable the dialog that asks for permission to use your camera and microphone (but remember the implications of this, you probably don't want to keep it that way after testing!). Enable this setting:

  • media.navigator.permission.disabled

Some example pages for WebRTC APIs are here:

Note that as the code only just landed, there are going to be bugs. Some known ones are the permissions dialog (if enabled) only remembering the first choice you make, giving you audio or video, but not both, and when disabling permissions, we always take the first camera, which is probably the one you didn't want. On some devices the video might be upside down, too. Expect most of these to be fixed in the next days and weeks. If you find any bugs that I haven't listed here, feel free to file in Bugzilla and we'll get to work on them.

Wednesday, February 27, 2013

Why Twitter pictures are sideways

We set out to do a demonstration of using WebRTC on an Android device before this years Mobile World Congress. I'm happy to say we made the deadline and were able to demonstrate some nice desktop Firefox to Android video calls. We only got the demonstration code working shortly before the deadline, though. (Actually, about 1 hour after we decided that we'd likely not make it, I got to tell everyone to disregard that because we just did, for a great "stop the press" moment). So here's a story about how we got hung up for almost a week on something that looked quite trivial at first.

The WebRTC stack in Firefox for Android is based on the upstream This codebase is shared between Firefox and Chrome, but as Firefox is the only one of those two to use the Android parts so far, we sort-of expected quite some work to do before getting anything working. That didn't end up being so bad: we hit a bunch of problems with the gyp buildsystem, another bunch were related to us running all the code off the main thread. One nice example that bit us is that running FindClass off of the main Dalvik thread doesn't do what you want. But overall, we found that we were able to get some audio and video running quite rapidly. Not perfect, but certainly good enough for a first demo.

That is, aside from one small issue: the video was sideways in portrait mode.

Experienced Android developers who are reading this are surely snickering by now.

The underlying cause of this is that the actual cameras in some devices are rotated compared to the phone shell. You're supposed to probe the Android Camera stack for information about the orientation of the camera, probe the phone's orientation sensors to know how the user is holding it, and rotate appropriately.

If you do a Google search for "Android video rotated" or "Android video sideways" you're going to get a lot of hits, so we knew we weren't the first to see this problem. For our use, we're specifically capturing data from an Android Camera object through a preview Surface. There are several threads on this issue on StackOverflow, some of them trying to cheat by pretending to be in landscape mode (a hack that didn't seem feasible for Firefox), but most of them conclude the issue must be solved through using the setDisplayOrientation call.

There are several downsides to this proposed solution: it is API Level >= 8 only, for starters. Given that we have dropped Froyo support (the final WebRTC code will likely require at least Android 2.3 or later, and almost certainly Android 4.0 or later for the best experience), that's not all that much of a problem for us, even more so because Camera support on Froyo or older devices is quite spotty anyway. But it makes me wonder what Camera apps on Froyo are supposed to do.

The second problem is that setDisplayOrientation only changes the orientation of the preview Surface. Now, for a normal Camera app that wants to show the user the video he's capturing, this is fine. But that's not what we're doing. We don't want to show the preview picture (in Java) at all! What to do with the data captured through WebRTC getUserMedia is entirely up to the webpage, which means it's to be handled in JavaScript (Gecko), and not the business of the Java layer at all. Maybe the JavaScript doesn't want to show the local video at all, for that matter. So our application doesn't actually want to display the Android preview, and so rotating it isn't going to make any difference. We just want the raw video data to send off to the video encoder.

Now, to go off on a small tangent, it turns out that not showing the video preview on an Android device isn't actually reliably possible pre-ICS. We currently use a dummy TextureView, which is known not to work on some devices, and will use a SurfaceTexture (which is guaranteed to work) on Android 4.0 or newer devices.

The Android Camera API also has a setRotate call, which might rotate the image data, or it might just put an EXIF flag that the image was rotated. This can only work if you're taking in JPEG data or if you're lucky and your phone actually rotates the data. None of the phones we wanted to use for the demo (Galaxy S3 and Nexus 4) do this. Taking in JPEG is out of the question, imagine the overhead going NV21->JPEG compress->JPEG decompress->I420->VP8 compress for every frame at 60fps! For the same reasons, we discarded the idea of doing the rotation with SurfaceTexture, setMatrix and letting the GPU do the work, because that would require a double NV21<=>RGB conversion as well.

Somewhere around this point, it was starting to dawn on me that this is the reason that pictures I'm taking with my Android phone come out sideways when posting on Twitter. The camera is actually capturing them sideways, and something along the chain misses that EXIF flag, which causes the picture to be displayed as it was taken. What makes this sad is that you can actually rotate JPEG data the right way in a lossless manner, so this is entirely unneeded. Or maybe they could just have made it easier for the camera apps to rotate the data before saving. Crazy idea, I know.

Okay, so here we are. We're getting in the raw camera data, and the only format we can rely on being available on every device is NV21 (except for the emulator, which will use RGB565, because this is Android after all). So, we need to rotate the NV21 data before sending it off towards our WebRTC video codec. We create a Bitmap with that image data and then rotate it using the Bitmap functions, right? Well, that would be fine, but unfortunately, even though NV21 is the only format you can rely on being generated by the camera, there is nothing else in Android that can understand it (there's, which "helpfully" only allows JPEG output). The only way to deal with is to write your own conversion code in Java.

Because we have a C++ backend, and the part of that includes libyuv, we ended up passing the data to that. libyuv has optimized code for YUV-style data handling (including NV21), so that's nice, though the implementation in the WebRTC code is buggy as well. At least there were some hooks to request the rotation included, so someone down there must have known about this madness.

So we only need to work around those know bugs, and voila, we have rotated our image. Easy, uh? Android is great!