BazelCon 2018 Day 1: Lightning Talks


JAMIE WAHLIN: My
name is Jamie Wahlin. I am a software engineer
at TCN, and today I will be speaking about
connecting applications with Protobuf and Bazel. First off, I want
to introduce to TCN. TCN is a leading provider
of cloud-based call center technology. Our products include inbound,
outbound, speech IVR reporting, and workforce
optimization features. We’re based in the unlikely
location of St. George, Utah, and it’s just an hour
away from Zion National Park, which is pictured here,
and yes I have done that hike. It’s terrifying, but
I highly recommend it. TCN started out with
a mono-repo in 1999. We had a small team, a
rapid development cycle, and the mono-repo worked
wonderfully for us for a few years. Our code base grew,
and we started facing complicated dependency
management issues, long build times, and out of
memory errors, which forced us to migrate to
multiple repositories. In 2016, we adopted
Protobuf and gRPC for the fourth version
of our product. As our number of
repositories has multiplied over the past couple of years
and our number of languages has increased, it has
become increasingly difficult to manage our protos. These are some of the
challenges our team has faced. We found it difficult to
keep applications in sync. For example, if a developer
updates a proto file on one repository and there are
10 others that depend on it, they have to update those
10 other repositories, and if they forget, which
they would sometimes, there would be
unexpected failures. We dealt with mismatched
proto bindings. We didn’t generate any
front-end proto bindings. We struggled with multilingual
compatibility issues in our protos, and the
files were scattered across multiple repositories. So it was time
consuming to track down our contracts and expectations. These challenges were negatively
impacting our development velocity and needed
to be addressed. At the beginning of this
year, a few team members and I investigated Bazel, and
began to transition to it. We started off by migrating
individual repositories to Bazel. This included generating
proto bindings with gRPC using Bazel’s existing
rules and also their custom rules, which
we needed for our two in-house protoc plugins. Just as a side note, these
two plugins are really cool. One generates a persistence
layer for Go from a proto file. This one’s open-source. There’s a link at
the end of the slides if you want to check
it out, and the second generates a Redux state
from a proto file, effectively bridging
the gap between protos and our application state. Next, we consolidated
our proto files into their own repository. This was made possible
by Bazel’s ability to generate proto bindings
from multiple languages and by its caching mechanism. Next, we ensured that our
front-end end applications remained in sync
with our protos. In order to do this,
we adopted TypeScript, and utilized two of
Improbabls’ open-source tools, namely protoc-gen-ts
and grpc-web. We were able to reliably
generate these proto bindings for TypeScript because of
Bazel’s standardized sandbox environment. We now benefit from
compile time errors if our front-end
applications are not in sync with our protos. Next, we moved to the mono-repo,
went back to our roots. This was made possible
by Bazel’s fast builds, remote caching, and support for
all the languages in our stack. We faced several
challenges along the way. We discovered there was a
big learning curve initially. Some rules were not mature
at the time we needed them, and our team had to,
sometimes forcefully, adjust to new tools and
development processes, and we even had a SAS
controller that melted. That was exciting, and this
was on the machine that was running all of our CI builds. We think it was due
to too much I/O. So we’ve limited the number of
cores each Bazel build can use, and so far, so good. [LAUGHTER] Now, think back
to the challenges our team faced with the protos. Bazel helped us,
directly or indirectly, tackle all of these challenges. We now generate proto bindings
for every language in our stack without complicated scripts. We’ve consolidated all our proto
files into a single directory. So we know exactly where to look
for contracts and expectations, and we’ve transitioned
to a mono-repo, which has synchronized
our applications and streamlined the
generation of proto bindings. Overall, Bazel
has made it easier for our team to
connect applications with protocol buffers. Thank you very much. [APPLAUSE] JINWENG CHEN: Hello, everyone. I’m Jin. I work on the Android
rules in Bazel, and today I’ll be giving
a status and overview of the Android rules in Bazel. So, what’s available now. What you can actually
use now, the Android rules to integrate
with the Android. SDK, NDK, they are
repository rules, but the way these are all
native, these are all Bazel. To build apps, you have the
Android binary and Android library rule. To import Android AARs,
you have aar_import rule. To run emulator
and device tests, you have the Android
instrumentation tests. To run local electric tests,
you have the Android local test, and then to build
cache [INAUDIBLE],, you have the
Android device rule. It’s fully integrated
for Android Studio. The plugin was released
open-source two years ago, and this works on
Windows, Mac, and Linux. So as a summary, this
is the complete matrix of what we support right now
with the exception of Android instrumentation test device
and building apps on RBE. Everything is supported
on all three OSes. So, performance. We have been doing a lot of
work on performance recently, and we have addressed
bottlenecks and achieved up to like 2.5 times [? speedup ?]
in the Android resource processing pipeline
using persistent workers. We also improved things like
Dexing, [INAUDIBLE] Dexing, integrating AAPT2, and
decoupling and the Android resource processing pipeline. Yeah. So we achieved quite a
variety of improvements here. So there’s a Bazel command
called bazel mobile-install. This is an alternative
to bazel build, and then ADB
installing your app. So this gives you this fast
iterative development workflow, and internally we
have benchmarks, and the red one is
for mobile-install. It’s a lot faster, essentially. So the Android rules
feature Java 8 desugaring, which allows you to use Java
language features like lambdas and Java 8 libraries
like Streams on Android. There’s a blog post
by Kevin Bierhoff. You can check this out
for more information. We have heard a lot about
remote build execution today, and I’m happy to share
that Android builds still work on RBE as well. Yeah, and on the right,
you can see the new JSON profile generated from
Bazel, created by Tobias, and I think, yeah, and this
is actually in Chrome tracer. So you can actually see the full
trace of your build in Chrome. Kotlin rules. The community has created
rules for building Kotlin, and they recently added support
for building Kotlin for Android using the
kt_android_library rule, and this is available in
the rules_kotlin repository. What we’re working
on right now, we are actively migrating the
rules of the Bazel server into Starlark. This decouples the
release cadence, and reduces the Bazel size. It’s available in rules_android. Mobile Install v2. We are actively rewriting
Mobile Install into Starlark, this is into a faster, more
incremental version of it, and it works with AAPT2. More work on performance
improvements. We are trying to come
up with a benchmarking framework for tracking Bazel’s
Android build performance over time. Emulator testing on
RBE– this is a big one because right now if
you want to launch nine emulators in
a local machine, you need a very beefy machine. If we can do that on RBE,
that would be amazing, like 100 [INAUDIBLE]
at the same time with minimal configuration. So no changes to existing build. Future work– we want to improve
our dependency management. Right now it’s a bit manual. So, on that. Migration support–
we want to create tooling guidance and
documentation for moving Android builds to Bazel. Faster Android Studio
plugin releases. We are going to refresh
our rules into– for example, we want
to support dynamic apps using the android_module rule. We want to split
up building of the, the build logic into
android_application and android_library, and finally
we want to also contribute more to the Kotlin rules. That’s it. That’s it. If you have any questions, feel
free to approach me or my TL, Alex in the back. That’s it. Thank you. [APPLAUSE] IRINA CHERNUSHINA:
Hello everyone. My name is Irina, and
I work in Bazel team. Let’s speak about C++ project,
which uses Bazel as the build system. Very probably that project will
use some third-party libraries, and very probably those
third-party libraries will not use Bazel, but
will use some other build systems such as CMake
or configure-make. The question is, how can
we build such project? One way would be actually bring
those third-party libraries to Bazel. This requires reverse
engineering of the existing build scripts, which
is a non-trivial task. It takes time and doesn’t
scale well with many libraries. The other way is to
actually run the builds of those external systems
as part of our build, and bring results to Bazel. If you create a rule for
each external system, then all the complexity
of the link visit will be inside the
rule, and so the work for bringing each dependency
in our project will be minimal. This scales well and
gives user the choice of gradual adoption of Bazel. This is what Bazel
team is working on now. This is a GitHub project,
of course, in Starlark. So let’s have a
look at the example and how it can look
in the BUILD file. Let’s say we have
cmake_external rule, which we supply with the
sources and dependencies, and then we can posit
target into the cc_library or cc_binary. Some design highlights–
since there are so many other build systems
in the world, we do not want to
build individual rules, but we want to create it
as a Starlark framework, easily expandable. We model the build of the
libraries as a black box. [INAUDIBLE] configure
and execute steps inside in one Bazel action. As it is the normal
Bazel action, we would benefit from
caching, and since you do not change the versions
of your libraries that often, the cache values is
going to live relatively long. We want the user to have the
full control over the external build. That’s why we provide
the possibility to pass any options
to the external system and also add some parts to
the underlying shell script. Our rules should
interoperate with Bazel, read Bazel settings and C++
toolchain so that the user doesn’t have to define
the same thing two times, and we also need to access
C++ information of file dependencies and provide
such information. The good news is it is
already available in Starlark in Bazel 0.17. We already have
CMake rule ready, and we are working on
Autotools and configure-make. Our next plans also include
remote execution support. This is our GitHub repository. We would be happy for your
feedback and contribution. Thank you. [APPLAUSE] GREGG DONOVAN: Hello. I’m– uh, wait. The speaker notes are– I’m going to be speaking
to you about why to use rules_kubernetes
and rules_docker to deploy your Bazel applications
to Kubernetes, and how it works to give
you fast, correct deploys. I’m also going to offer five
general Kubernetes with Bazel tips all in five minutes. Kubernetes is smart enough to
deploy only what you change, and it does so by hashing
its inputs, just like Bazel. These hashes include
both the YAML and the exact versions of the
containers that you’re running. Taking advantage of incremental
deployment in Kubernetes requires that you
can deterministically generate all of your inputs,
including your containers. rules_docker provides rules
for generating container images deterministically,
allowing you to build a single, repeatable
package from base container to your own application code. rules_kubernetes builds on this. It takes, as input, a
set of Docker containers in an input_kubernets YAML
with container reference placeholders. It builds your
containers, pushes them to the Container Registry, gets
the SHA-256 of each container image, and then replaces
those placeholders with the resolved
SHA-256 references. This approach really shines with
a multi-application mono-repo. When you change shared
libraries, Debian files, tooling, et cetera,
you need to know which components need to be deployed. In a simple application,
you might be able to keep this in your head. But at scale, you
need automation. With this approach, we avoid
two problems– under-deploying, not deploying change components
leaving them unvalidated on master, and
over-deploying, waiting for potentially
expensive rollouts despite not having made changes. We can radically
simplify our CI/CD flow. We can have a single pipeline
for building, testing, and redeploying all
of our applications, relying on Bazel to
negotiate what actually needs to be deployed
with the Container Registry in Kubernetes. In order to understand
how to take advantage of what Bazel gives
us, it’s worth diving into how it works
at a low level. How does Bazel negotiate
this reconciliation with the Container
Registry in Kubernetes? So just, what is a
Docker container? It’s an ordered list of tarballs
representing file system changes. Additions, edits, and
deletions– each layer is represented as a SHA-256
of the uncompressed contents. The overall image is a SHA-256
of the layers and the metadata. Rules_docker builds these
layers with Starlark and Python. Not having to run the
Docker clients or the daemon simplifies built-in
CI, removes the issues with building Docker images
from within a container, and allows Bazel’s parallelism
to speed container layer building. The standard
Container Registry v2 API provides caching
from the local build to the remote
Container Registry. Rules_docker uses
HTTP head requests to check for an existing SHA-256
of an already-cached container image in the SHA-256 of
each container layer. Finally, Kubernetes deployment
hashes the pod template to determine if your application
needs to be redeployed. If the pod template
hash is unchanged, there is no need for a
rollout, and with this you have the benefits of Bazel’s
repeatability from source to deployment, and
now for the tips. Many of Kubernetes’ applications
don’t need a full operating system, and can
simply package up their direct and
runtime dependencies. For Java, that’s glibc,
SSL, and the JVM. The Distroless project
provides simple base image, and rules_docker provides
rules to utilize them, but sometimes you do need a full
operating system and a package manager. In that case, the aggregate
Debian package rules from bazel-toolchains provides
transitive Debian package resolution in a deterministic,
Bazel-friendly way. There are many
Kubernetes workflow tools besides rules_kubernetes. Whichever one you use,
prefer SHA-256 references for first-party code. This is supported by default
in Skaffold, for example, and most of them will still
allow you to build your docker images via rules_docker. Most Kubernetes
users will eventually face the wall of YAML problem– how to tame thousands of
lines of configuration. Instead, consider using
the Kubernetes client APIs rather than YAML for
complex applications. You get IntelliSense,
compiler support, and other first-class language
tools lacking in YAML. The Pulumi project is a
bit of an inspiration here. It allows you to create cloud
resources, projects, clusters, buckets, et cetera,
and Kubernetes objects with real programming
languages, not DSLs. It’s a refreshing alternative
to YAML and other limited configuration
languages, and finally, if you’re using
Docker and Kubernetes, you may also be using
Grafana for graphing. My colleague at
Etsy, Kevin Gessner, has written Bazel
rules for creating parameterizable Grafana
dashboards with Python, and installing them in a
Docker container running a properly configured Grafana. Thank you for coming, and please
contact me at the conference if you have any questions. [APPLAUSE] LAURENT LE BRUN: Hi. I am Laurent. I have been working on Starlark. So, Starlark is a language you
use for avoiding BUILD and bzl files, and I will
give you an update for what we’ve been working
on about the language and mostly about the tooling. So if you’ve ever heard the name
Skylark, just forget about it. It’s called Starlark. OK? So we’ve been working on trying
to simplify the build API. So the big focus was on
simplifying those and making them simpler to use,
and so simplify the API. And so our goal is to make
sure that everything is stable. So every function should
be properly documented with nice examples, easy to
use or we should remove it. So we are not there yet. We’re still working on
it, and so our focus is to get stability
before beta 1.0. So this is just a list of
things we’ve been working on. For example, the first
one ctx actions.args. So if you write Starlark rules,
you might want to use it. It will allow you to want to
create more efficient command lines. Data configuration
has been removed. So that’s also a
nice simplification if you’ve seen that before,
and a few other things. So we have also a separate
repository for Starlark with a complete
language specification. So we have also been working
on a lot of the tools. So one of them is Buildifier. So it has been
around for some time, but it can now format bzl files. So this is production ready. We are using it at Google. We have reformatted all
our 50,000 bzl files, and it’s [INAUDIBLE] check. So it’s working. On top of that, we
have a Buildozer. So there are not many
news on this front. It’s just that– so if you
want to edit your BUILD files automatically, so either
you have lots of them or you want to do
that from a script, you can add or
remove dependencies easily with a command
line interface. You can also use a
Go API if you want to do more complex
code transformation, like we name a variable
across the enter code base, these kind of things. this is also production ready. We have been
working on a Linter. So we already had a Linter. So we are working
on a replacement. The code has been
submitted to this. You can try it. It might work or maybe soon,
and so having a Linter is nice, but most of the
time you don’t want to manually [? wade ?] all the
warnings and fix them yourself. So we are providing a new
command with Buildifier to automatically apply fixes
to your build and bzl files, and this is important, as
we’re making changes to Bazel. So incompatible
changes, and you want to keep your code
working with Bazel. Also, this has been
submitted today. We are going to work on it. Contributions are
welcome because we can add more automated fixes. We have been working, also,
on documentation generation. So there was a
tool called Skydoc. It was not working well. So there’s a replacement
that is coming soon, like in the next few months. That should make
things much easier for people who
write Starlark rules and want to provide good
documentation for your users. And we have a debugger. So typically, when
you have complex code, you might want to debug
it and have a breakpoint and execute it line by line. So when you have
complex Skylark code, you will be able
to use a debugger. So there is a VS
Code plugin that will be submitted in the
next few months as well, and [INAUDIBLE]. And finally, in IDEs, what you
really want is code completion, like you start
writing your bzl file, and you want to see what
functions are available, and code completion– here are more details. [APPLAUSE] ANDREA BICAN: Hi, everyone. [INAUDIBLE] Thanks Laurent
for introducing my project. I’ve been working in the Bazel
theme for the last four months, and I’ve been working
on integrating Starlark in an IDE, all those nice
features that can speed up your coding process. So, first and foremost, the
project is not yet open-source. The plan is to release it at
the beginning of next year. So you guys can try out,
and see for yourself how all the features work. OK, so when I say
all the features, I mean the following four. I mean code completion, go-to
definition, signature help, and diagnostics. All those four features
are implemented in the language
server, which means that it has behind it the
language server protocol, which means that you can plug it in
in any IDE that supports it. So you can have
the same features and same implementation in
VS Code in Eclipse and VM in Emacs and a few others. So, first and foremost,
is code completion. So that was the first
feature that I implemented, and I think the
coolest thing about it is that, in a language like
Starlark, which is new and not yet widely-known, I
think it’s way easier to learn it because it
gives you nice suggestions. It gives you
relevant suggestions. It gives you context-aware
suggestions, and– for example, you have
all the loaded symbols. You have function
names, parameter names, everything that you declare
inside a file, and also all the built-ins and I think
it’s pretty nice to have the built-ins because
you don’t have to go back and forth between
your IDE and the specs and the syntax and
the documentation. You have everything there. So you have, in
context, actions. You don’t need to know that
there are three functions. You know it because it’s there. So, you know the name. You know the required
parameter, the return type, and also the official
documentation, and then what we’re doing is
that trying to kind of improve the completion with
some type inference, and we base this on some cases. And the first one is– we look at var, and by looking
at the right-hand side, we know that’s a
string, and then knowing that var is string, then we can
understand that the greeting is a string, too. So then, knowing that
greeting is a string, then we can actually give you
suggestions that are relevant. So all the building
functions for string. Then, this is a nice one. What we’re doing, we’re
basing this on the assumption that programmers are decent,
and they follow conventions. So what we’re
doing is that we’re looking at the doc
strings and, in this case, we can understand
that the parameter cfg is of type configuration
just because you specified it. Then we have path
completion, and I mentioned that we have
relevant suggestions. So when you’re
inside a load path, we actually give you suggestions
for paths that you can load, and then once you
have the loaded path, then we give you
suggestions for symbols that you can load
from that path. And the next one is
go-to definition, and, you know, it’s pretty nice
to just jump around in a file and see, where was this defined? In the global scope or
inside a function body? But I think the nicest use case
is when you load something, and when you use it, it actually
jumps to the definition. So it opens the file and places
your cursor where it’s defined. So you know what you’re using,
and it’s signature help, and signature help is
this text box that you have above your cursor when
you’re inside a function call, and it just tells you
the many parameters that you have to fill in
when you call the function. And the last one is
diagnostics, and diagnostics are the result of calling
Buildozer, and what we’re doing is just to render all the
warnings and all the errors in the UI so you can
just catch them early. Yeah. That’s been my project. Those are all the features
that will soon be available. So just check it out,
and make sure you’re trying once you’re out there. [APPLAUSE] JODI MORAN: Hi everyone. My name is Jodi. So, in my experience, making
developer feedback loops shorter is win-win. It makes your code better,
and it makes you happier. So, why don’t existing
commercially available CI tools go further to give you
high-quality feedback faster? The project I’m
going show you today is my sketch of the ideal CI. I had two goals in mind. First of all, I want to give you
feedback as fast as possible. The feedback should
be fast enough so that you can stay in
flow while writing code and not have to do
any context switching. Second, I want to give
you better feedback. For example, I could tell you
before you push to the upstream whether or not your changes
will integrate with it. I could also tell
you before you push to the upstream how
much of an impact your changes are going
to have on the system. I decided to build this
CI specific to Bazel rather than supporting
many build tools because I think it will provide
a better developer experience. Bazel’s speed is, of course, the
foundation for fast feedback, and it’s extensive ability makes
CI for multi-language systems simpler, but perhaps
most importantly, Bazel gives me access to
the abstract graph structure of the build, which
allows me to write much better feedback than
would otherwise be possible. OK, enough introduction. Now let’s take a look at what
the project actually does. It consists of a command line
client that you run locally and a remote server
that’s private to you. Every few seconds, the
client detects any changes that you’ve made locally
that you haven’t yet pushed to the upstream,
and it sends those to the remote server. On receiving those
changes, the remote server first pulls from the
upstream and builds it. If the build succeeds,
your local changes are integrated into the
upstream using a rebase, and finally, a build is
run on the integrated code. Feedback about the
process is continuously communicated to you so that
you’ll find out immediately if there’s any problem, like
that the upstream doesn’t build or that there is
a merge conflict. At the end of the loop, you’ll
know that your changes are good to push to the upstream. Now let’s take a look at the
UI in little bit more detail. At the top, I’m showing
the current status. Here, I show you how many
files you’ve changed locally. Here, how many targets
directly include your changes. Here, I’m showing you
the number of targets that are downstream of the
targets that are directly including your changes,
and this gives you some idea of how much
impact the changes are going to have on the system. Here, I’m showing you
the number of targets that are leaf targets
that are downstream, and those are the ones that
get passed to Bazel test. Below the information
about the timing of the build, I’m showing you
information about the success so far, which includes
how many targets have been built successfully
and how many tests passed. At the bottom of the screen,
we show the Bazel logs. Now let’s look at the
project in action again. Again, the idea is that this
tool is running continuously in the background, automating
the process of pulling from the upstream, integrating
your changes in building so that as soon as you’re
ready to receive feedback about your changes, the feedback
is already ready for you. You can stay in flow
while writing code, and still get great quality
feedback about your changes. Right now, this is
just a small prototype, but I can think of lots
of interesting ways it could be extended. For example, I could let you
know if someone else is editing the same files that you’re
editing before either of you has pushed to the upstream. If you haven’t pushed to
the upstream for a while, I could give you a reminder to
do that to share your changes. I can also give you
more information about the risk of the
changes that you’re making, for example, if a
file that you’re editing has a history of
breaking the build. I’d also like to give
you even faster feedback. For example, I could
run the tests that are most likely to break first. I can let you know information
about which parts of the build are running slow so
that you can optimize them. I can also let you
know even about tests that aren’t providing any value
because historically they’ve never found any bugs. That’s all that I have time
to talk about right now, but if any of you are interested
in these kind of ideas, I’d really love to
talk about it further. Please find me around the
conference or drop me a line. Thank you. [APPLAUSE] MATHIEU BOESPFLUG: Hi, again. So what I’d like to
talk about today– just on this very short talk,
I’ll try to be brief, is Bazel, you’ve heard a lot
since this morning, it talks a lot about
fast and correct. Let’s accept fast, but I
want to challenge correct. [LAUGHTER] First of all, I want
to challenge correct, and secondly, Melody, this
morning, talked about the fact that, at Google, there were
2 billion lines of code in the biggest mono-repo,
and so something that’s really important when you
have 2 billion lines of code, and it’s important for us
too even though we don’t have nearly that amount
of code, is to really never rebuild the world. And so I want to talk about
how we at Tweag use Bazel, and try to achieve full
correctness without rebuilding the world. So, how do we get correctness? Actually, a big
part of correctness is reproducible
builds, and to get reproducible builds you
basically need two really key ingredients. You need your builds
to be hermetic, and you need your builds
to be deterministic, and so if you have hermetic
and deterministic, those two properties together,
then you can hope to get reproducible builds. The problem is that I don’t
think you really quite get reproducible builds with
Bazel alone because, arguably, Bazel builds are not
all that hermetic. There’s plenty of
hermeticity holes in Bazel today, at least
if you use it out of the box in the
open-source version, and I don’t know how
Google uses it internally. But out of the box– so for example, if you’re
using the C++ rules or many of the native rules are
basically auto-configuring the toolchains– for example,
GCC, the C compiler– from the environment,
from the host. If the environment
changes, what happens then? Likewise, your
builds will depend on a bunch of system libraries
like glibc, pthread, et cetera, et cetera, and also,
in practice, a lot of the rules that are implemented
in Starlark, they often use the run shell
action, and sometimes they– so there’s an
option that you can pass through run shell
that basically says, don’t wipe the environment. So, for example, I
can call, basically, from that action any binary
that happens to be on the path, and so that’s
hurting hermeticity. There’s a bunch
of workspace rules that can also hurt hermeticity. So, the typical solution that
people have to all of this is, does Docker help? And the problem with
Docker is that the build inputs are still
not completely captured by the build description. I challenge you, in
particular, to think about what happens if you
change the Docker file that characterizes the image that you
use to run all your actions in. That can change completely
independently of Bazel, and Bazel doesn’t know
when those changes happen. So you can get some
reproducibility problems there. The other problem is the
Docker images are not always reproducible– and this
is a known problem– from a Docker file,
and Docker images don’t really compose well. Ultimately, there are no
fine-grained dependencies in Docker layers, and so
this means that there– as I said in the beginning,
you either have a choice. You can either somehow track
the changes in the Docker files, but then any time anything
changes in the Docker file, you have to rebuild the
entire world, right? Because anything in Bazel
can depend on anything from the environment
provided by the Docker file. Or you can just say, all right. Too bad. Forget about hermeticity. So I want to– what we do at Tweag
is that we actually use another system different
from Docker, but very similar, which has the same
ideas in Docker. So for example,
you have a notion of derivation, which is
essentially akin to Docker layers, but the difference
is that derivations are much more fine-grained. We have a notion of
expressions, which are kind of like Docker
files, but the key there is determinism and
reproducibility. And so because we have
fine-grained dependencies, targets in Bazel depend on
specific layers in Docker parlance or specific
Nix derivations rather than just
the whole image. And so what this ultimately
means is that your rebuilds– whenever you add a
derivation, for example– your rebuilds are
basically pretty minimal. So for example, if
I just change zlib, then anything that
depends on zlib will rebuild, but nothing else. So Nix is pretty cool,
and so we added– we have an experimental rule
set called rules_nixpkgs that you guys can use. We often use– when we
use Bazel in production, we nearly always use it
together with rules_nixpkgs. Thank you for your time. [APPLAUSE] KLAUS AEHLIG: OK,
so I’m Klaus Aehlig, and I’m talking about Bazel
and external repositories, and by external repositories
the main interest is sources of truly
third-party dependencies, and more precisely I’m talking
about the question which versions of these external
dependencies you get. That all started
with a discussion at last year’s BazelCon, and
there was kind of some, well, discussion on–
there was a version that roughly looked as follows. So, imagine you take a
project to a fresh checkout, and you look at
the workspace file, and it’s very declarative. It shows you which
repositories to depend on, which are the relevant branches,
and what transformations to do, and so on, but there are
no specific versions. So it’s abstract description of
what are your external source dependencies, and
that is something that has to be changed
very rarely only if you’re adding a new dependency
or removing one, and then you build,
and you build at the latest-known
good snapshot, and then you check out an
old version of your project, and you build against the
version of your dependencies that were used at that time. So, how did we imagine
that even to work? We weren’t thinking about
ways of doing some magic, but basically the idea was
ignore the workspace file, and use another file,
and don’t worry. You don’t have to write
that file by hand. It’s generated from
the workspace file in a controlled way,
and it got committed. So it contains the precise IDs,
commit IDs instead of branches, and, for good measure,
also timestamps so that they can
shallow clone, and it contains the hashes of the
sources that you generated. So even with all
your transformations, and maybe not having the
same version of these tools during the
transformations, you know that you have the
right code, or at least you get notified if you
did something wrong. And actually it’s just a
Starlark value contained in that file, so the build
can just load that file, and do whatever it
wants with these values, like adding precise
snapshots of the dependency and the version string
in a functional way. OK, so the idea is
you have a file that contains all the relevant
information of the versions, and, how would you
think of updating that? The idea is to have a
new command, bazel sync, and then it looks
roughly like that. You see some
information saying, oh, I was executing the
rule google_protobuf, and this is the
commit I get, and that was the timestamp and so on. So it executes the workspace
file unconditionally fully. It executes all the rules
in the workspace file, and sees what comes out, and
it records all the versions that the rules returned, as
this is the precise version I got when following the
branch master in a file, and the idea is then
to test and commit that file to the repository,
and since that file is not just the Starlark value, but a
nicely, canonically-formatted Starlark value, you also
get meaning for this. And the good thing is
that is reality now, well, as of Bazel 0.19, but
you can already download the first RC, but don’t worry. We haven’t changed the
evaluation mechanism of Bazel completely. It’s fully opt-in,
controlled by the RC file. So it just [INAUDIBLE]
some options. It looks much scarier
than it actually is. So there’s one
option telling Bazel, well, whenever you execute
an external workspace rule, record the return
value and the fact that you executed the rule in
a file, and you name the file. So that is what you
want to do for sync. Sync, by its nature,
executes all the rules in the workspace file, and then
if you write that down you’ve got the values, and
it tells you, well, for build you have
another option. We’ll say, actually,
when building, ignore the workspace file. Use that other file instead
to know what the rules are. And then you can say, well,
actually use the hashes. Verify the hashes of
the external repository. This is the file where
we get all the hashes, and then you list all
the rules where you want the hashes to be verified. For example, you might
not want to verify that you get the same
CC configure output, and actually it
looks a bit redundant to have the same file in
there are over and over again, but that can also be a feature. Remember, it’s just a
Starlark value in that file. So you can add another
file that includes that value and that does some
transformations to the recorded values, like if you know,
for my Git repository, also have a way to
download tar archives for specific snapshots. So all that
transformation knowledge can go into a separate file. OK, so that is the
present, and for the future we have some ideas, but
more importantly, we want your feedback–
what is needed, and where we should
go from there. So please talk to us either
in person today or tomorrow or by email. Thanks. [APPLAUSE] XIN GAO: Hello, everyone. My name is Xin. I’m from the remote execution,
[INAUDIBLE] team from Canada, and today I’m going to
introduce you a Bazel container. So a container image is a
lightweight, standalone, executable package
of software that includes everything you
need to run an application, and in this presentation I
might use the word container and image interchangeably, but
for both I mean a container image, and so the Bazel
community has been asking for a Bazel container
for a while, and our team builds containers
for remote execution, and we are also the
maintainers of rules_docker and rules_kubernetes. So we found ourselves
in a position to provide a Bazel
container as well. So we built it. So, it allows people to
try out Bazel very easy with no configuration,
no installation. Just install Docker, and it’s
very good for CI as well. So the Bazel
container is publicly available in the Google
Cloud Marketplace, and you can pull the container
at this Container Registry path. So here’s what the
landing page looks like, and containers with
different Bazel versions are kept available and pullable
from these container tags. So in this container,
we install a– first it’s based on a
Google-maintained Ubuntu 16.04 base image, and then we install
a Google build and tested Clang with sanitizer support, and
in addition we have OpenJDK 8, and two versions of Python
installed to support Bazel’s Java and Python builds. So there are
several key features of this Bazel container. First, containers with
different Bazel versions are kept available, and
you can just pull them. You can choose which
Bazel version you want, and in addition,
to make building with sanitizers very easy, we
have a default bazelrc file in this container. So if you like to build
with asan, tsan, or msan, you just need one extra flag. So initially, when we
come to build containers, we use Dockerfile,
and we found ourselves copy paste too much code. So to install the same
tool or same compiler in multiple containers, we
just copy paste the same code in multiple Docker files. This is very error-prone
and very inefficient. So we take advantage
of Bazel and Starlark. So we developed a
set of Starlark rules to construct Docker
images using Bazel. So instead of specifying
the files to add and the entry point or the
command of the container in Dockerfile, you specify
them in Bazel build targets, and here are the rules, and they
are in bazel-toolchains repo, rules_docker, and also
in base-images-docker. You can Google them afterwards
and these are all links as well. So here are two examples. So with this rules, it allows
us to reuse and compose layer targets to
construct complete images. So the toolchain
container rule is, you can think of that as like
you create a complete image, and the language tool layer
is a image that represents a very specific– only contains
a specific tool or a group of tools. So in the language
tool layer rule, you specify what
packages to install, and what inbound variables
to set for this layer, and some other files to add,
and in the toolchain container rule, you just specify a
list of language tool layer targets in the toolchain
container rule. So with these, you can easily
create toolchain container targets by composing
existing layer targets. So the one on the left
is a simplified version of the Bazel container,
toolchain container targets, and the one on the right is just
like a container foo target, and they share the same Java,
and Python later targets. But just to note that we are
not reusing the actual layer contents of the file yet. So if you first build a
Bazel container on the left, then you build a container
foo on the right, you build a container
from scratch. So in the coming
quarter, we are planning to work on fully automating
the release process, and integrating it into the
Bazel’s own release process, and then we are also working
on making the computer generation fully deterministic,
and, most importantly, we would like to make
the actual layer contents reusable from cache,
not just the targets. So that means if you
build the container foo on the right in
the previous slides, after building the
Bazel container, then you will reuse the Java,
Clang, and Python layer files, and say if you use multiple
containers that share common layers in multiple
Cloud Build steps on GCB, the Google
Cloud Builder, then those common
layers will just be pulled once
and cached by TCB. So stay tuned, and join
the bazel-discuss group at this link, and this
container is also documented in the Bazel website. So you can check that out later
on this link, and that’s it. Thank you. [APPLAUSE] JAY CONROD: Hi, everyone. My name is Jay. I work on the Go tools
language team here in New York. I’m the maintainer for
rules_go and Gazelle, and today I’m going to
tell you about Autogazelle. So just a quick show of hands. How many people here
use Go in any capacity? Cool. A lot of hands up. All right. Here we go. So, Gazelle is a
command line tool that generates build
files for Go projects that follow traditional
Go conventions. It also works for
Protocol Buffers. So here’s a build file
that Gazelle generated for the Kubernetes project. Everything you see here
is generated by Gazelle, except for the lines in italics. Gazelle allows you to make
manual changes to build files, and it’ll preserve them. After you run it, it’ll
update build files in place. You see go_library, go_binary
here, a bunch of dependencies. It works pretty well. Kubernetes uses this to
manage over 2,600 build files. So, in general, it’s
pretty successful, but coming from
the Go ecosystem, people don’t want to
write build files at all. Generally, you can just
like go build and go run, and it works, and, why do I
have to write this extra thing? And it’s especially
annoying that you have to run this extra command. You do your editing, and
then you run bazel build, and it’s like, oh, I
forgot this dependency. It’s not there. So I have to go back, and
I have to run Gazelle, and now my build file works. So this is where
Autogazelle takes over. This removes that extra step. So, I have a quick demo for you. This is from the
Kubernetes repository, and before I started, I deleted
the go_rules out of all the build files, and I deleted
this build file entirely from command kubelete, but
we’ll run a Bazel build anyway. We’ll see what happens. Autogazelle kicks off. It takes a few seconds to
start up the first time. There it goes. It’s all done, and we
start a main build. It’s in the cache already. So that finishes quickly. All right. Our build succeeded. Let’s see what’s
in the build file that Gazelle generated for us. All right, so we have our
go_library and go_binary. This is the same
build file as before. Because I deleted it, it
doesn’t have the manual changes. Let’s make some
changes of our own. So we’ll open up kubelet.go,
and we’ll add new import. Normally, this
would be something that you have to update
the build file for. We’ll save that, and while we’re
at it, let’s rename the file. I never could figure
out how to pronounce it. Is it “cube-lit” or “koob-lit?” I’ll just rename it to
“kooblet” with two Os. [LAUGHTER] We kick off our build. Autogazelle kicks off again. It runs a little
faster this time, and we have to
recompile and relink. This is a big binary. So it takes a few seconds. All right. There we go. Let’s see what our build
file looks like now. It looks like it picked
up our new source file name and our new import. All right. [APPLAUSE] My button’s not working again. Can we go to the next slide
if anyone has manual control? I’ll just proceed. So, how does that work? We have a Bash script, and it’s
installed in tool/Bazel, and– thank you– the Bazel
that’s in your path when you run a Bazel command is
going to look for this script, and if it finds it,
it’s going to execute it instead of the real Bazel. So this script is going
to run Autogazelle, and when it’s done, it
will execute the real Bazel with your original
command line arguments. So Autogazelle is a program
with two pieces, a client and a server, and
the client just connects to the server
over a Unix domain socket it uses for synchronization. It doesn’t actually
exchange any messages. The server watches your
file system for changes, and whenever it
accepts a connection it runs Gazelle in those
specific directories that changed. So you get an incremental run. Gazelle runs much
faster because it only runs in specific directories
rather than the whole repo. So, for a large repo
like Kubernetes, that can save you
several seconds. If you’re curious to
learn more, check out the bazel-gazelle repository. This is checked into
cmd/autogazelle, and just a warning– this is like
super new and experimental. So maybe don’t quite use
this for production projects yet, but feel free to try it
out and file issues, and let me know if you have any questions. Thank you. [APPLAUSE]

Leave a Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Copyright © 2019 Geted Tabs Online. All rights reserved.