[RFC] rump hypcall integration for LKL by thehajime · Pull Request #255 · lkl/linux

thehajime · 2016-10-28T12:46:44Z

(since this is RFC, I'd be happy to have any comments and suggestions from yours)

This PR provides an initial implementation of rump kernel hypercalls (*1) which brings us a couple of interesting features to LKL.

The highlight of new features are:

additional host environments, NetBSD, qemu-arm, qemu-x86_64, xen other than linux/freebsd/windows which current LKL supports.
- execuse: I haven't tested that much, only tested with qemu-arm, qemu-x86_64 on Linux.
non hijack-based existing application support with a dedicated libc bind to LKL using musl libc
- rumprun/frankenlibc unikernel integration
new thread primitive based on green thread (a.k.a. userspace thread, related to official usermode thread support in LKL #250 ?)

The addition of rump hypercalls does not interfere the current implementation of LKL, but with additional external repositories (*2), you will run LKLed applications with rump hypercall.

Non-hijack application support requires to build an (existing) application by dedicated build toolchain (cross compilation as rumprun does), but it will solve most of hijack limitations such as conflicted symbols/namespaces, non exposed symbols (e.g., replace hidden symbols in glibc). With this toolchain, I confirmed nginx, netperf, and ghc (a haskell compiler) somehow work fine, though those need more work to be more than just a hello world.

The green thread implementation is obviously useful to avoid the expensive task of context switches. I saw some of netperf benchmarks outperform pthread based one (will re-measure with netperf in my spare time).
(rump hypercall has pthread-based thread implementation but I didn't test it in this PR).

I'm going to upstream the external repositories (*2) as well as this PR so, this PR will be a base of all of external dependencies.

I think merging non-master branch of LKL and synchronize periodically with master branch is also reasonable since the changes (of this PR) are a bit large.

*1 rump kernel
http://book.rumpkernel.org

*2 additional (external) repositories

rumprun unikernel: https://github.com/libos-nuse/rumprun
frankenlibc unikernel: https://github.com/libos-nuse/frankenlibc
buildrump.sh scripts: https://github.com/libos-nuse/buildrump.sh
musl libc LKL support: https://github.com/libos-nuse/musl

This change is

Rondom · 2016-10-30T16:05:57Z

 {
 }

+void read_persistent_clock(struct timespec *ts)


It might make sense to make this public i.e. in addition add some helper function in user space using settimeofday/adjtimex that syncs the host time with the lkl time. One possible use case is syncing the clock with the host in case of drift in a long-running application or a simple jump of the host time due to manual change. I know everyone can write this quickly themselves, but it would be a nice convenience function.

thanks for the comment @Rondom.

It might make sense to make this public i.e. in addition add some helper function in user space using settimeofday/adjtimex that syncs the host time with the lkl time.

I don't understand this comment; what do you mean by making a function public ? btw, 66513c9 exposes all kernel functions by putting a prefix to avoid symbol collision.
(see #17 (comment) more detail)

though you might be aware of this, this additional function overrides a function of kernel (below), which each architecture can define its own way to access to a clock.

void __weak read_persistent_clock(struct timespec *ts) in kernel/time/timekeeping.c.

I was not aware (and not really awake while writing that answer). Thanks for the explanation.

Rondom · 2016-10-30T16:07:04Z

 CONFIG_ISO9660_FS=y
-CONFIG_BTRFS_FS=y
-CONFIG_BTRFS_FS_POSIX_ACL=y
+#CONFIG_BTRFS_FS=y


What is the reason for disabling btrfs?

this is because this LKL kernel stops during make test, though I need to investigate more what's the root cause.

https://circleci.com/gh/libos-nuse/frankenlibc/136

I was suspecting some of busy loops may cause the stuck.

tavip · 2016-11-01T08:17:06Z

Awesome ! This looks good at a first glace except two minor issues.

One is the BTRFS issue. Do you see the stuck issue when running w/ rump or with regular Linux backend? If it is under rump/qemu you can probably start gdb and see where it is looping. On the other hand I think that we perhaps should look into adding proper IRQ support, since we have at least one new target that supports it.

There is another concertn I have with regard to the lkl__sync ops, there might be slower then the __sync primitives and that could cause some performance regressions.

thehajime · 2016-11-02T00:21:41Z

One is the BTRFS issue. Do you see the stuck issue when running w/ rump or with regular Linux backend? If it is under rump/qemu you can probably start gdb and see where it is looping. On the other hand I think that we perhaps should look into adding proper IRQ support, since we have at least one new target that supports it.

https://gist.github.com/thehajime/f8af72bf625e90d397c2584a285d97d3

it stopped at do_xor_speed(), jiffies is not updated at all during the loop (I guess we fixed this before).
I will look at if the non-preemptive scheduler (with green thread implementation) works badly.

There is another concertn I have with regard to the lkl__sync ops, there might be slower then the __sync primitives and that could cause some performance regressions.

in the current version, I can't find proper atomic ops (at least can't use __sync prefixed atomic ops) on arm-none-eabi target so I needed to wrap the ops. x86_64 target should work as before without any performance regressions.

In any case, since atomic ops seems to not have a portable interface, I thought it's a good idea having wrapper functions (or adding host_ops entries).

liuyuan10

The LKL_HIJACK_DEBUG doesn't work anymore for me. Can you also rebase to master so that I can check the performance of it?

liuyuan10 · 2016-11-01T23:50:06Z

 	lkl_thread_t tid;
 	struct task_struct *prev_sched;
 	unsigned long stackend;
+	void *rump_client;	/* for syscall proxy */


addr of stackend should be the end of thread info?

though I couldn't find how stackend is used, I agree with you by changing the place of new member.

liuyuan10 · 2016-11-02T00:47:13Z


 	ret = snprintf(sysfs_path + sysfs_path_len, sizeof(sysfs_path) - sysfs_path_len, "/%s/block",
 		       virtio_name);
-	free(virtio_name);


Is it safe to do so? seems like d_name is pointing a memory that will be freed in lkl_closedir

this should be fixed now with replacing strdup by lkl_ops.mem_alloc + memcpy.

thehajime · 2016-11-02T07:13:24Z

The LKL_HIJACK_DEBUG doesn't work anymore for me.

should be fixed now. thanks !

Can you also rebase to master so that I can check the performance of it?

I have a trouble with a rebased version on qemu-x86_64 (on linux and rump/frankenlibc are working fine). It seems like the direct syscall patch (e1cd839) doesn't like me - revert the commit with a hello world program works fine.

(#257 didn't help me btw)

you can try a measurement with hijack library - this should work fine now.

liuyuan10 · 2016-11-02T17:11:55Z

Yes. The performance looks neutral. The direct syscall requires support of long jmp. Maybe you want to check if it works as expected on qemu?

thehajime · 2016-11-19T10:19:12Z

The direct syscall requires support of long jmp. Maybe you want to check if it works as expected on qemu?

setjmp/longjmp seems to be working on qemu. incomplete pthread implementation outside of LKL might be an issue with the direct syscall implementation (e1cd839). so #267 didn't help either.

I will split some of commits which is independent from rump integration, while investigating the above issue separately.

tavip · 2016-12-15T09:32:09Z

Can one of the admins verify this patch?

This will allow us to reimplpement atomic ops with lock/touch/unlock way if underlying hosts don't support a particular atomic ops. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

This only handles signals right after a system call was invoked. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

This allows us to define IRQ related operation outside of the kernel by host_ops implementation. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

This host_ops entry will provide a way to pass any variables (e.g., environmental variables) to an LKL application from the host environment. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

This commit adds an implementation of new host_ops by using rump hypercall (wrapped as an LKL host_ops) provided by rumpkernel project. The rump hypercall implementation is specifically integrated with the one provided by frankenlibc project, which is an alternative way to use rump hypercalls. There are other rump hypercall implementations such as rumprun, NetBSD kernel tree, but this commit only focuses on the frankenlibc integration. The new features covered by this commits ranges in various ways, but here is some highlights. - New thread primitives frankenlibc implements user-space thread implementation based on makecontext(3)&co with non-preemptive (cooperative) scheduler implementation. - LKL specific standard library bind frankenlibc offers cross-compiler build chains to build an LKL application bound to LKL-versioned standard library (i.e., libc). Our musl libc implementation is provided by different repository. So, any application can be built with this toolchain to use LKL instead of host libc. - New host environments support LKLed application now has potentials to run on the platform which rump hypercall implementations support. frankenlibc supports Liunx, FreeBSD, NetBSD, and qemu-arm. Others supposrts run an LKL applications on hypervisors (kvm, xen, etc) The frankenlibc code is located at the following repository, but it will be upstreamed to the original one (justincormack/frankenlibc). https://github.com/libos-nuse/frankenlibc Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

qemu-arm support with frankenlibc & lkl is limited at this moment: no network (virtio-net) is not supported yet, etc. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

The code is not built as-is but put an implementation using rump syscall proxy which makes an LKL application communicate with outside of the process. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

This commit offers an integration with rumprun unikernel. It is only tested with qemu-system-x86_64 but it potentially works fine with x86 baremetal machines as well as xen hypervior. An initial PCI device support is also implemented by relying on exposed virtual devices from hyperviros: virtio-net device is tested for the moment. The rumprun repository to support lkl build is not upstreamed yet - you can try with the following repository before it will be in. https://github.com/libos-nuse/rumprun Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

This reverts commit 2c507e4 ("device core: Remove deprecated create_singlethread_workqueue"). This is a temporary fix for qemu-arm hangs on the following command that schedule_work() triggers (?) dead-lock that nanosleep on hello->main() never returns. $ qemu-system-arm -M versatilepb -m 512M -nographic -serial null \ -semihosting -kernel rumpobj/tests/hello This should be fixed in a transparent way in a future. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

copumpkin · 2017-08-09T03:11:28Z

This looks cool. What's the status on it?

thehajime · 2017-08-13T01:36:22Z

@copumpkin I've constantly updating this branch. 2 PRs were also submitted to rumpkernel but still no immediate upstream yet.

laijs · 2017-11-15T08:30:07Z

I'm curious how the signals be delivered to the rump-client, cooperated lkl app or hijacked app?

thehajime · 2017-11-15T23:30:29Z

I'm curious how the signals be delivered to the rump-client, cooperated lkl app or hijacked app?

Did you mean the rump-client as syscall proxy from external processes ?
If yes, then signals are not delivered to the client: only input/output of syscalls are proxyed/delivered to the client.

If not, would you care to elaborate ?

laijs · 2017-11-17T03:16:27Z

Sure, I meant the rump-client as syscall proxy from external processes.
Thank you and I understood the signals are not delivered to the client.
But why the signals need to be handled in the kernel? since it can't be delivered.

thehajime · 2017-11-20T02:26:36Z

But why the signals need to be handled in the kernel? since it can't be delivered.

this rump hypcall integration has several modes of execution: external process via syscall proxy is one of them. musl libc integration with no dependency to host kernel is the other one.

I am specifically trying to support alarm(3) used in netperf with this signal related patch.
Unlike hijack library mode as of current LKL, alarm(3) calls go to an LKL instance, wait for a signal (SIGALRM) but no delivery before this patch. (In the hijack mode, it is delivered by the host since alarm(3) goes to the host)

So, we need to have some sort of do_signal() and calling it at a proper point. it seems that calling this in run_syscall() is not the best place: any suggestions are welcome.

Is that clear to you ? Let me know if not.

gparmer · 2018-04-09T17:37:09Z

Is there a blocker on this? We have a rump hypcall-based environment, and would like to use Linux, but I've been waiting to put time into lkl until this was pulled in. Should we stop waiting?

gparmer · 2018-04-18T16:44:59Z

+
+	if (write) {
+		if (size == 1) {
+#ifdef __x86_64__


I'm guessing this means that x86-32 is not supported?

LKL does not support x86-32, yet #341

this is a messy part of lots of ifdefs and a result of what i personally tried. this can be expanded a platform which is currently not supported (e.g., x86-32).

gparmer · 2018-04-18T16:50:05Z

+
+static void *rump_tls_get(struct lkl_tls_key *key)
+{
+	return rumpuser_curlwp();


This doesn't get the TLS variable given the key, correct? Knowing rumpkernel's handling of TLS (where gets are implicit on __thread variables), it seems like there might be a mismatch between the LKL and rumpkernel models here. I like explicit calls to get TLS based on a key in LKL much more.

I suppose the assumption here is that LKL only uses TLS to store the currently active thread?

as you can see, this function drops the key argument the result only indicate a TLS val, not localized by a key.

I suppose the assumption here is that LKL only uses TLS to store the currently active thread?

Not really, just a limitation of in this RFC stage.

gparmer · 2018-04-18T16:58:19Z

+	.mem_free = rump_mem_free,
+	.ioremap = lkl_ioremap,
+	.iomem_access = lkl_iomem_access,
+	.jmp_buf_set = jmp_buf_set,


I know this is the LKL way of doing this. Perhaps this isn't the place to ask this, but why not include a more abstract function that simply switches to another thread. Could be implemented with setjmp/longjmp, or with OS primitives below this layer.

(asking these question here should be fine with me)

I'm not sure if I understand your question. maybe the commit 79131f2 is relevant to this question ?

@gparmer I have an old branch that removes most of this low level ops and switches to higher level opts like alloc/switch/free contexts, see: https://github.com/tavip/linux/commits/the-expanse, in particular tavip@e182fe0.

The reason I did not merge this is that we have optimizations to reduce the system call latency (see e1cd839) which I could not map well with the new scheme.

Thank you both for the information and references! @thehajime @tavip

I think that the patches @tavip referenced for host context switching do what I was contemplating. I didn't foresee the complications with the system call path (nor do I fully understand the interaction now).

From my perspective (i.e. wanting to not use lkl as a library, but as a normal resource manager and abstraction provider, while using host abstractions such as threads and memory management), some of this seemed a little overly complex. However, given the goals of the system, this is all making quite a bit more sense. It seems really quite well engineered.

I don't mean to distract lkl at all, but I wonder if a number of simplifying assumptions would simplify the design. For example, if I could make the following set of assumptions:

all host threads that will make "system calls" into the library are created using the thread creation APIs of lkl (with the exception of the initial thread -- that executes main),

all interrupts are executed as host threads that the lkl creates and controls (thus execute from the Linux perspective more like the interrupt threads in the real-time patches), and

the context switch API focused on switching host threads.

Could it enable "direct" system calls (without switching stacks/contexts)? Would it allow preemptive execution? SMP?

Again, sorry for distracting from the main lkl goals and thrust. I certainly understand that lkl will not adopt these assumptions, given its goals. However, I'd very much value your opinion if these assumptions could simplify some of the interactions with the host system.

gparmer · 2018-04-18T17:01:24Z

+	return ret;
+}
+
+struct lkl_dev_net_ops rumpfd_ops = {


The LKL way of doing things is to export the raw hardware I/O to the underlying/hosting system. In the BMK rumpkernel implementation, you actually use the devices within the RK for I/O. Given that you've implemented the PCI rumpuser interface, are LKL device drivers supported?

this particular piece of code (rumpfd_ops) is used by https://github.com/libos-nuse/frankenlibc, which is a variant of rumprun unikernel. and you're right: with rumprun bmk LKL can use drivers/net/virtio-net.c or drivers/block/virtio-blk.c. I have tried e1000 driver of Linux with qemu e1000 emulation over LKL. Though recently rumprun LKL is not well maintained (I hope I do as early as possible).

I'm familiar with rumprun (we have our own hacked up version with a backend for our own OS). I was not sure that you'd be able to run native drivers (e.g. e1000) since I didn't see the virtual to physical rumpuser function (which is used in bmk, for example to enable devices to understand how to interact with DMA).

Great to hear that devices are at least possible, if not supported!

I was unaware of the mainlining initiative so I certainly understand why this patchset is currently unmaintained.

thehajime · 2018-04-18T23:58:29Z

Is there a blocker on this? We have a rump hypcall-based environment, and would like to use Linux, but I've been waiting to put time into lkl until this was pulled in. Should we stop waiting?

@gparmer thanks for the comment (and sorry for the late response). My personal opinion on this PR is still not changed: I would go for upstream basic LKL part to linus tree first, then add more extensions like this PR after that.

But for the mean time, it would be of course great to receive any comments/suggestions/patches :)

(ref: #304)

Rondom reviewed Oct 30, 2016

View reviewed changes

liuyuan10 reviewed Nov 2, 2016

View reviewed changes

thehajime force-pushed the rump-hypcall-upstream branch 2 times, most recently from a40346e to 78522ae Compare November 2, 2016 05:38

thehajime force-pushed the rump-hypcall-upstream branch from 78522ae to a0dd66c Compare November 7, 2016 05:40

liuyuan10 mentioned this pull request Nov 16, 2016

Ask for suggestion of building a liblkl.so without libc #264

Closed

thehajime force-pushed the rump-hypcall-upstream branch from a0dd66c to fff885e Compare November 19, 2016 10:15

thehajime force-pushed the rump-hypcall-upstream branch 2 times, most recently from 51eca75 to 76abb0b Compare November 25, 2016 09:03

thehajime force-pushed the rump-hypcall-upstream branch from 76abb0b to 291ceb8 Compare January 12, 2017 04:42

thehajime mentioned this pull request Jan 19, 2017

Proposal: have cptofs support a simple script file that instructs it on what to copy into/out of an image #299

Open

thehajime mentioned this pull request Feb 9, 2017

lkl: add LKL_HIJACK_SYSCTL to configure sysctl values #319

Merged

thehajime force-pushed the rump-hypcall-upstream branch 7 times, most recently from 3ae4248 to 7d95a54 Compare February 12, 2017 23:50

thehajime added 4 commits February 13, 2017 08:53

lkl: let atomic ops outsourced

970ebee

This will allow us to reimplpement atomic ops with lock/touch/unlock way if underlying hosts don't support a particular atomic ops. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

lkl: add signal delivery support

12ea48f

This only handles signals right after a system call was invoked. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

lkl: add irq_{request/release} host_ops entries

f5ad407

This allows us to define IRQ related operation outside of the kernel by host_ops implementation. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

lkl: add getparam host_ops entry

5a87063

This host_ops entry will provide a way to pass any variables (e.g., environmental variables) to an LKL application from the host environment. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

thehajime added 6 commits February 13, 2017 08:53

lkl: fix arm-none-eabi- build with frankenlibc

499ba64

qemu-arm support with frankenlibc & lkl is limited at this moment: no network (virtio-net) is not supported yet, etc. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

lkl: add experimental rump syscall proxy support

75bae62

The code is not built as-is but put an implementation using rump syscall proxy which makes an LKL application communicate with outside of the process. Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

lkl: work around for arm build failure (Issue lkl#229)

5f9d302

Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

lkl: temporary disabled btrfs

f82b294

Signed-off-by: Hajime Tazaki <thehajime@gmail.com>

thehajime force-pushed the rump-hypcall-upstream branch from 7d95a54 to f82b294 Compare February 13, 2017 00:11

thehajime force-pushed the rump-hypcall-upstream branch from ea9039a to 95be33a Compare February 13, 2017 08:27

thehajime mentioned this pull request Mar 6, 2017

switch to idle_host_task when idle #336

Merged

gparmer reviewed Apr 18, 2018

View reviewed changes

Uh oh!

Conversation

thehajime commented Oct 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rondom Oct 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tavip commented Nov 1, 2016

Uh oh!

thehajime commented Nov 2, 2016

Uh oh!

liuyuan10 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thehajime Nov 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thehajime commented Nov 2, 2016

Uh oh!

liuyuan10 commented Nov 2, 2016

Uh oh!

thehajime commented Nov 19, 2016

Uh oh!

tavip commented Dec 15, 2016

Uh oh!

copumpkin commented Aug 9, 2017

Uh oh!

thehajime commented Aug 13, 2017

Uh oh!

laijs commented Nov 15, 2017

Uh oh!

thehajime commented Nov 15, 2017

Uh oh!

laijs commented Nov 17, 2017

Uh oh!

thehajime commented Nov 20, 2017

Uh oh!

gparmer commented Apr 9, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gparmer Apr 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thehajime Apr 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thehajime commented Oct 28, 2016 •

edited

Loading

Rondom Oct 30, 2016 •

edited

Loading

thehajime Nov 2, 2016 •

edited

Loading

gparmer Apr 18, 2018 •

edited

Loading

thehajime Apr 18, 2018 •

edited

Loading