Scala World is over for this year - thanks for following along. Hope you’ve enjoyed our coverage - see you again next year!

Here’s a list (in no particular order) of all the slide decks from the conference that I’ve found online so far:

• Move Over Free Monads: Make Way for Free Applicatives!
• Dotty: Exploring the future of Scala
• The Need for Async
• A deep dive into scalac
• scala.meta tutorial
• Scala and the JVM for Big Data: Lessons from Spark
• Hands-on TLC
• Practical Binary with Scodec and Shapeless

Over in the other room, Erik Osheim (non, @d6) is giving an insightful talk about lessons learned from the Spire project and some of the design decisions behind cats. He’s very passionate about building a community and encouraging newcomers to try functional programming.

Mathias & Johannes have just done a really beautiful demo, showing a chain of three participants - a server generating a stream of log data, an Akka HTTP server (reading that data & spitting it out as a processed stream), and curl, reading the processed data. All three were happily playing their roles, scrolling through data, until Mathias paused curl. Using netstat, he was able to show us how the buffer on curl’s incoming network socket filled up over the course of a few seconds, causing the middle Akka HTTP server to respect that back-pressure and stop processing data, causing it’s own incoming buffer to start filling, until a few seconds later it exerted back-pressure on the first server, finally causing that server to stop generating content. In order to get this kind of beautiful effect to work, every layer needs to be able to propagate back-pressure.

Quite amusingly, the Akka HTTP team have been looking at the logs of their own little Maven repository (repo.spray.io, where they publish all their own Spray artifacts) and have found that they get pretty high traffic, 98% of which is 404 Not Found. People are adding repo.spray.io as resolver in their sbt resolver chain, which means that it can get all the artifact-requesting traffic from those projects, including, for instance, private group and artifact ids. They can learn some surprising stuff from that data...

And of course, instead of returning 404, we could just craft our own jars and return them for other requests!

A cackle of admiration at the potential for unmitigated evil goes around the room. Have I mentioned that I really don’t like adding additional resolvers to my projects? It slows the build down, makes you dependent on repos which might not be there next year... and leaves you open to maliciously crafted jars!

Might be a time to remind people that sbt-pgp allows you - in principle - to verify the PGP signatures of all the jars used by your project, which could help combat this kind of awful evilness:

Akka http is the successor of Spray, that we use quite extensively in the architecture of the Guardian Content API. The team released a 1.0 version, which is not yet optimised for performance.

Reactive Streams send data from producers to consumers, but consumers can indicate that they are full, and don’t require any more data through ‘back-pressure’ - which is actually the consumers lack-of-demand (ie a consumer actually has to indicate they want data to get it).

Mariot and Roberto are now settling down for the last talk of the day - ‘Reactive Streams & Akka HTTP’ with Mathias Doenitz & Johannes Rudolph - pretty much the entire development team are in this room! Mathias Doenitz has just asked if how many people know what Reactive Streams are. This is my answer:

...I think there’s something involving back-pressure involved as well, I’ll let you know what I learn.

I watched a demonstration of Idris, the dependently-typed language, by its creator, Edwin Brady. He showed us a few examples of how you can do the same kinds of calculations on types that you would normally do on values, and how the types can guide the programmer toward writing the correct program in a process that he calls “type-driven development”. He was even able to ask Idris to fill in the easier parts of the implementation automatically. Edwin also has a book out. I’ve read the first couple of chapters and I’m now hooked on Idris!

After that, Lars Hupel talked about proof assistants, demonstrating two tools called Isabelle and Leon. Leon is a proof assistant for Scala that makes use of the standard Scala `assert` and `require` statements to define formal specifications.

In a gif-tastic presentation, Pascal and Julien introduced their library Précepte. It’s a toolkit based around the State monad that makes it easy to pass state around a pure functional program for logging, metrics, etc. They made a lot of references to the Free monad, which is without a doubt the catchphrase of the conference. It’s everywhere.

Debasish Ghosh’s talk was about modelling domains in a functional rather than an object-oriented style. In traditional OO domain modelling we tend to focus first on defining the domain entities in detail, then moving on to defining the behaviours of and interactions between those entities. But in the case of functional modelling, we define an “algebra” for the domain, consisting of pure functions, abstract types and laws. All this sounds very abstract, but he explained it well using his standard example of securities trading. He’s writing a book about this stuff, which looks like a worthy addition to any functional programmer’s bookshelf.

It really is a lovely day here in Penrith. Perhaps the next talk could take place outside...

Konrad’s talk was packed with detail about performance, including both theory (lock-free vs wait-free algorithms) and practical tips (benchmarking with sbt-jmh). It was sad to see how many slides he had to skip because he ran out of time.

Now I’m listening to George Leontiev (@folone) talking about Typelevel. This project started with a controversial fork of the Scala compiler about a year ago, but I haven’t heard too much about what the Typelevel team are doing since then. Cats, the scalaz competitor, has become popular recently, but I’m not sure exactly how that project relates to Typelevel.

Next up is Konrad Malawski, replacing Viktor Klang who had to drop out at the last minute. He’s talking about how to write high-performance asynchronous code in Scala.

Miles is doing that thing again where he renders Daniel Spiewak speechless. This time, Miles is informing Daniel that it is, in fact, possible to create a refinement of Nothing. Daniel declines to repeat the details for the video, on the grounds that it is literally too awful to preserve for posterity.

Daniel’s speaking now about a reset of Scalaz-streams - which apparently has pretty awful performance - with a better, more constrained model in FS² :

Elegance is stupid! Stupidity is elegant!

This seems to be a common theme at Scala World, following on from Rúnar’s talk yesterday. Daniel Spiewak also has great speaking energy, as you can see.

For those of you playing Spiewak bingo, he’s already said both “and here’s where we enter bizarro land” and “this is really really really cool”.

The conference is taking place at the Rheged Centre on the outskirts of Penrith. It’s an amazing venue, maybe even better than the Beurs van Berlage in Amsterdam, where Scala Days was held this year.

It’s hard to describe the Rheged Centre, but it’s like a medieval fortress that a Bond villian has repurposed as his lair. Here’s how it looks at night.

“Laws allow other people to forget things about your code - where ‘other people’ may be defined as your past self” - Daniel, talking about applying mathematical laws to software.

Yesterday Miles presentation was a really special event. Miles described how he has tried, for a long time, to solve what looks like a very difficult type level problem. Unfortunately I am unable to explain this problem, but I can report to you what happened when he tried to explain the solution he finally found.

The audience reaction was a long puzzling silent, only interrupted, by Daniel applauding slowly, literally amazed, by the approach and its effectiveness.

In this auditorium, filled with smart scala developers, only few understood the problem miles has solved and less were able to evaluate the solution he provided. Everyone was staring at the screen, trying to ‘mind compile’ the type level code they were reading.

It’s a gloriously sunny day here in Penrith. Perfect for sitting in a big dark room and talking about abstract syntax trees! Last night we had a delicious dinner followed by an inscrutable lecture by the inimitable Paul Phillips. Was he trolling us? Who knows.

Today we have a keynote from Daniel Spiewak and plenty of interesting talks about Dotty, formal proofs, Idris and more.

Interesting fact from the sponsor message from Goldman Sachs: they have 130,000,000 lines of code in production just for risk evaluation and pricing.

Hang on, there’s just been a pitch invasion - Miles was just getting in his ‘Typeclass Derivation for Cats’, and a canine interloper ran across the stage... no word yet on whether Miles will be allowed extra time to cope with this intervention.

Michael Pilquist’s talk ‘Practical Binary with Scodec: Applying Typelevel Programming and Shapeless to the Mundane’ has just finished, and I think the entire audience was blown away. It looks like the most fun way possible to create codecs dealing with binary data. Wish I’d had this back when I was trying to write some scratchy code to talk over USB with my old Garmin GPS.

After ruminating on Adriaan’s talk for a while, and discussing it with Eugene Yokota during the break, it’s starting to make more sense. The slides are available here. If we think as a type alias as the type-level equivalent of a `val`, then the idea is to introduce a type-level equivalent of a `def`. In other words, a function that takes in one or more types as input and returns a type. He gives an example of pattern matching on types to calculate other types, which reminded me a lot of the sort of type-level calculations you can do in Idris. The question about whether this foray into so-called “dependent types” is a good thing for Scala is what proved to be controversial during the talk.

This diagram of the dependencies between the scala.meta modules is as beautiful as the “traits mixed in to Global” diagram from my scalac talk earlier.

Roberto has gone to learn about Scodec while I hear about scala.meta from its creator, Eugene Burmako. Eugene was also the lead developer of Scala’s macro functionality.

scala.meta is a generic toolkit for parsing and generating Scala programs, both at the syntax (source code) and semantic (trees, types, etc.) level. The idea is that it will act as a platform for IDEs and other tooling, as well as language features such as macros. The first release, 0.1, will not include macro support, but it have enough features to enable a wide range of tooling.

scala.meta is currently at version 0.0.3, but people are already using it in production! For example, Andrea Lattuada gave a talk at yesterday’s unconference about using scala.meta to extract API documentation by parsing a web application’s Spray routing directives.

Adriaan’s talk turned out to be a pitch for SIP-23, amongst other things. He advocated radical changes to Scala’s implicit resolution mechanism, introducing a new concept that he calls “opaque types”. Only about three people in the room fully understood it, but they certainly felt strongly about it. Miles Sabin’s first reaction to the idea was a long, heartfelt sigh.

Bill’s new collection library looks really interesting, and hearing about the design decisions behind it was very valuable. I’ll have to give it a try. I couldn’t find the source on GitHub, but it must be available somewhere.

Now Adriaan, the tech lead of Scala at Typesafe, is giving us an update on the compiler team’s recent work and roadmap.

“Use needles not machetes” - Bill Venners explaining a strategy for getting people to upgrade as you deprecate and then remove functionality, offering the Python 2 -> Python 3 transition as a counter example. A little bit of pain with small steps - people will complain as they upgrade, but they will do it. On the other hand, when the transition is a massive amputation that takes off your leg...

Chris and I are ready to listen to ‘Simplifying Scala Collections’ by Bill Venners, starting now, followed by Scala Implicits 2·0 by Adriaan Moors...

My talk about hacking the Scala compiler to make it do silly things went off without a hitch. The slides are here. Now Simon Ochsenreither is doing a live coding session, showing how to make much more useful changes to the compiler. I’m happy to report that thanks to the research for my talk, I actually understand what he’s talking about.

Speaking about ‘lubbing’, Eugene Yokota speaked yesterday about the sbt plugin he started to turn it off. The fact that is is possible to hack quite easily with the language, even by side-effect, is a hidden feature that I loved.

Chris has just introduced a Scala compiler plugin to cross-compile Scala - like Scala.js - but instead of JavaScript, it compiles to BASIC. Applause from the audience. It’s right here on GitHub - come and get it!

Our very own Chris Birchall is giving ‘A Deep Dive into Scalac’ right now - really glad to learn a bit more about the AST... you can see I got confused about this at the weekend:

Chris has highlighted the Typer as being the really expensive phase of Scala compilation - it’s doing things like the afore-mentioned ‘lubbing’, often a really expensive computation.

Rúnar Bjarnason gave a great keynote this morning, ‘Constraints Liberate, Liberties Constrain’ - arguing that flexibility actually leads to limits in use, while constraints on component design lead to freedom and power when putting those components together into systems. Personally, I felt like jumping up and down in agreement at this point - Rúnar gave several examples of where this happens, but it resonated with me particularly because of my work with Scala-based Git tool the BFG. git filter-branch gives a wide range of extremely flexible options for rewriting Git repository history, but due to that freedom is unable to optimise it’s performance for the common use case of removing unwanted data - it can not make the same optimisations that the BFG can, to be 50-1000x faster. Constraints Liberate.

Joe Barnes is speaking about type level programming , or how we can expedite computation from runtime to compile time. You can enter its subspace, with slides and code examples, on its github repository. If you are interested by the approach you can as well read how to implement quicksort at type level by Julien.

Jon is doing some interesting type-level programming here - he’s contrived the types of the Area and Width & Height expressions so that it would be a compile-time error if he hadn’t handled every case in the ps.handle() expression. This works due to ‘lubbing’ - calculating the Least Upper Bound of the combined expressions. At yesterday’s unconference Eugene Yokota showed a proposal for a ‘-Yno-lub’ flag, disabling lubbing, arguing that the compiler could do much less work, and that the resulting behaviour would be more predictable for a developer. Here’s an example of where lubbing is giving the developer more predictability - arguably at the expense of abusing the Scala compiler.

“I haven’t shown the implementation because implementations are boring; we like types. Save value-level code for lesser conferences.”

After a high-quality keynote from Runar, we have a very sleep-deprived Jon Pretty telling us about dependent types. Jon is not only giving a talk, but also single-handedly organising the whole conference.

This looks a little like Skaro, the world of the Daleks, but it’s actually Scala World, the world of Scala developers. Very beautiful when you can see the view.

Welcome to our liveblog of the Scala World conference. After a soggy but enjoyable hike and an “unconference” yesterday, the conference proper starts today. Flaky wifi permitting, we’ll be liveblogging throughout.