What's Clojure's best usage ?

[Jan Goyvaerts]

Recently I've asked this Alex Miller about a typical use case for Clojure. He replied: none. It's for general purpose. Personally, I don't think so. Other people still have to be able to read and maintain it. :-)

So asking it here again: Are there cases in which Clojure excels compared to Java or Scala ?

I'd like to try it at work. But since I don't know it well enough, I'd settle for knowing the typical use cases. So when I stumble on them at work I can set Clojure at work without fear.

Many thanks !

Jan

[Sean Corfield]

Jan Goyvaerts wrote:Recently I've asked this Alex Miller about a typical use case for Clojure. He replied: none. It's for general purpose. Personally, I don't think so. Other people still have to be able to read and maintain it. :-)

As someone has been using Clojure in production for a year, I'll confirm what Alex told you: it's a general purpose language that works well wherever Java could be used. At World Singles we use Clojure for our persistence layer (wrapping JDBC and MongoDB), for email generation, for log file analysis, for interacting with our search engine (XML and JSON generation for REST interaction), and a lot of other general purpose stuff.

We find Clojure is a lot more concise than Java - and therefore provides less code to maintain.

[Jan Goyvaerts]

I won't argue the point. :-) IF I were the only one developing software at work. Or everybody here would be convinced about Clojure's added value. However, practical Clojure experience is currently far from common. That's why I would need a clear case to prove Clojure's added value to the locals. Especially the managers, who'd rather let you write more code in Java.

So, where's Clojure *vastly* better than Java ? So I can impress the skeptics here. So they'll see the world moved on since Java. :-)

[Palak Mathur]

I second Jan in this regard, why should I select Clojure in place of Java. Moreover, why shouldn't I go for languages like Scala (I do not have experience in either Scala) as far as conciseness is concerned. (I will definitely ask the same question to Scala specialists as well).

A new language is needed when the current languages do not handle certain scenarios effectively or the code becomes uneasy to handle or may such reasons. What were such reasons that Clojure was created? Some use cases as Jan asked would definitely help in this understanding.

[Chas Emerick]

Hi Jan! :-)

Alex and Sean are quite right; Clojure is a general-purpose programming language, so there is no "typical" use case for it. It's being used in domains as diverse as:

• everyday web programming (including replacing front-end JavaScript via Backtype, acquired by Twitter)


• generating and producing music (e.g. Overtone, demonstrated quite dramatically at the last Clojure Conj)


• automating computing infrastructure, from scripting local builds (with Leiningen) to one, two, or 1,000 compute nodes in the cloud or your own hardware (with Pallet)


• reverse-engineering the human genome (also demonstrated at the last Clojure Conj)

I could go on and on. Even a quick skim of the couple of Clojure "success stories" should convince you of its wide applicability.

But, you asked, "where is Clojure *vastly* better than Java?" It may be more instructive to ask, "where is Java as powerful Clojure?"

As someone that continues to derive the majority of his living from a product that is written and maintained entirely in Java (it was first released in 2003, thus the language choice), I can say quite honestly that there are very few points on which Java matches Clojure. The only area where Java exceeds Clojure is in tooling — where Java's verbosity and baroque type system demand very advanced editing and completion/visualization features in order to work productively. Thankfully, while Clojure tooling is hardly bad (I personally prefer Eclipse with the excellent Counterclockwise plugin — available in the Eclipse marketplace — and Leiningen outpaces Ant, Maven, and Gradle by a wide margin IMO), Clojure's typical concision and the far more powerful abstractions it provides makes whacking out reams of code simply unnecessary. This is the same reason why few people moving to Python or Ruby (or, Jython or JRuby) from Java find that they need an "IDE experience": once you get out from under Java, your dependence upon specific tool capabilities will drop significantly.

Now, if you're looking for talking points to help convince "the locals" (as you put it), I would focus on three points:

1. Functional programming (FP) and immutability. Clojure is definitely opinionated in guiding you to avoid mutation of state in place, and there is quite simply no downsides to that "policy". This is made possible by Clojure's uniquely capable persistent immutable data structures, which makes programming in an FP style efficient. Once you go FP, you'll never go back.


2. There is no better environment on the JVM for working with data, period. The "Java way" of hiding data within ill-suited class façades complicates the usage of that data by encouraging the production of tons of "little languages" (a.k.a. class APIs), which prevent one from readily accessing and manipulating the data within, or from readily composing the functions that operate over that data. Data really only ever comes in a few forms: maps, sets, sequences, a couple variants of each, and then all your various scalars. Clojure is a great simplifying force in that it makes it easy to work with these foundational organizations of data.


3. You can reuse all of your investment in Java and the JVM from within Clojure, and you can call code and libraries written in Clojure from Java as well; thus, any transition can occur over time, and you can maintain a mixed environment very effectively.

You could also mention macros, though they are much more subtle tools that should be brought to bear with discretion. In any case, we talk about all of these topics in the book, and compare and contrast Clojure with Java, Python, and Ruby throughout. ;-)

I hope that's helpful.

--
(coauthor of Clojure Programming from O'Reilly; creator of Clojure Atlas)

[chris webster]

Hi Chas,

You make some very interesting points on the benefits of Clojure, and as a developer, Clojure is high on my to-do list of topics to explore in more detail. FP is definitely interesting and potentially a very powerful alternative to current programming approaches, as Rich Hickey's Simple Made Easy talk illustrates, but that's also a significant challenge when FP + Clojure skills are still relatively rare. And your point about the reduced need for (and availability of) tooling in Clojure compared to Java also implies another subtle shift in common working practices.

So if you're working in a commercial "enterprise" environment with a significant existing investment in Java, I still don't see a persuasive argument for switching to Clojure, particulary if you are considering Scala as an alternative.

Clojure is inherently functional, so the language and the FP paradigm work very well together. But the language is also very different from Java, so developers would have to learn the new paradigm and the new language at the same time. In an ideal world this might well be preferable, but more usually we have to deal with people with different skills/experience making a challenging transition under pressure of project deadlines, existing workload etc. Admittedly, they can all buy your book - which looks great - but they still have to sit down and learn both the new language and the new way of thinking about their work.

Scala, on the other hand, is also an FP language but is far more obviously rooted in Java, which makes the transition from Java to FP with Scala more gradual. This is one of the factors mentioned by the team leaders at The Guardian newspaper when discussing their migration from Java to Scala: they found it was easier for existing developers to make the transition and to start taking early advantage of the benefits of the shift to FP in Scala while still being able to apply much of their Java knowledge from day 1.

What experience do you have - or have you gathered from other serious Clojure users - of making this transition in a large-scale environment, and what factors have you observed that can help improve the chances of success e.g. new application, small project initially?

I've only looked at Clojure briefly, and I like the elegance of the language and I'm intrigued by the shift in thinking required by FP. But my impression is that there are still relatively few large scale applications in Clojure, Clojure skills are rare on the market which implies problems with maintainability at least in the short term, and there is a sense that the language and its ecosystem has not yet matured to the point where it makes a realistic alternative right now for businesses with a large investment in existing Java technologies, so I'm not sure how many corporate managers would want to be early adopters for critical applications. FUD is a reality in many workplaces and it's not always entirely groundless.

Clojure still looks like a lot of fun to learn, though.

[Jan Goyvaerts]

I don't doubt the technical merits of Clojure. The STM alone is already impressive. To us, the technical people that is.

However big the merits, this will not do if I were to convince my hierarchical superiors why I should spend their money on a learning curve. "You already know Java, don't you ? So why bother ? " They're managers - they're not interested in whether a language is technically superior to another language.

HOWEVER ... if I were to tell them Clojure will allow me to develop a certain feature much faster, I'm sure they'll let me go for it.

So, is there something that comes to mind ?

[Sean Corfield]

chris webster wrote:And your point about the reduced need for (and availability of) tooling in Clojure compared to Java also implies another subtle shift in common working practices.

That's true. The most common workflow for Clojure development tends to be REPL-first, interactively "growing" the application in small pieces, and evolving production code and unit tests out of the expressions developed in the REPL. Groovy, Python, Ruby and Scala all support REPL-driven development but for those languages it's more of an added convenience than a fundamental building block in the development environment.

Scala, on the other hand, is also an FP language but is far more obviously rooted in Java, which makes the transition from Java to FP with Scala more gradual.

Whilst that's true for some teams, it's important to bear in mind that Scala is still primarily an OOP language with mutable data by default and you can write "Java" in Scala without leveraging any of the benefits of functional programming. Your Scala code will be more concise than your Java code but it will still be "the same".

But my impression is that there are still relatively few large scale applications in Clojure, Clojure skills are rare on the market which implies problems with maintainability at least in the short term, and there is a sense that the language and its ecosystem has not yet matured to the point where it makes a realistic alternative right now for businesses with a large investment in existing Java technologies, so I'm not sure how many corporate managers would want to be early adopters for critical applications. FUD is a reality in many workplaces and it's not always entirely groundless.

If a business has "a large investment in existing Java technologies" and hasn't shown much inclination to dramatically improve productivity, it's probably too conservative to adopt Clojure. Scala is trying hard to sell itself to "the enterprise" and, based on discussions I had at JAXconf last year with a lot of "enterprise" Java developers, I don't think those large, conservative organizations are even ready to adopt Scala in a way that brings the real benefits - they'll use it as a "better Java" but still write the same code, just in fewer lines - and they'll run into a skills shortage with Scala too (and, in my opinion, find that junior developers just don't do as well with Scala as more senior, more expensive developers).

There's no doubt that Clojure is very different. Companies that are looking for big productivity gains and are willing to try new technology - and mix different technologies on a single project - are more likely to be successful with Clojure.

[Palak Mathur]

Sean Corfield wrote:

chris webster wrote:And your point about the reduced need for (and availability of) tooling in Clojure compared to Java also implies another subtle shift in common working practices.

That's true. The most common workflow for Clojure development tends to be REPL-first, interactively "growing" the application in small pieces, and evolving production code and unit tests out of the expressions developed in the REPL. Groovy, Python, Ruby and Scala all support REPL-driven development but for those languages it's more of an added convenience than a fundamental building block in the development environment.

Scala, on the other hand, is also an FP language but is far more obviously rooted in Java, which makes the transition from Java to FP with Scala more gradual.

Whilst that's true for some teams, it's important to bear in mind that Scala is still primarily an OOP language with mutable data by default and you can write "Java" in Scala without leveraging any of the benefits of functional programming. Your Scala code will be more concise than your Java code but it will still be "the same".

But my impression is that there are still relatively few large scale applications in Clojure, Clojure skills are rare on the market which implies problems with maintainability at least in the short term, and there is a sense that the language and its ecosystem has not yet matured to the point where it makes a realistic alternative right now for businesses with a large investment in existing Java technologies, so I'm not sure how many corporate managers would want to be early adopters for critical applications. FUD is a reality in many workplaces and it's not always entirely groundless.

If a business has "a large investment in existing Java technologies" and hasn't shown much inclination to dramatically improve productivity, it's probably too conservative to adopt Clojure. Scala is trying hard to sell itself to "the enterprise" and, based on discussions I had at JAXconf last year with a lot of "enterprise" Java developers, I don't think those large, conservative organizations are even ready to adopt Scala in a way that brings the real benefits - they'll use it as a "better Java" but still write the same code, just in fewer lines - and they'll run into a skills shortage with Scala too (and, in my opinion, find that junior developers just don't do as well with Scala as more senior, more expensive developers).

There's no doubt that Clojure is very different. Companies that are looking for big productivity gains and are willing to try new technology - and mix different technologies on a single project - are more likely to be successful with Clojure.

For most of the enterprises, switching to a new language is a tough decision. In most cases the enterprises are not only using languages like Java but also using products that can talk with Java (for an example). For that matter, if they switch to a new language, then they also have to change the way their products interact with Java or go for a different product or develop such a solution on their own.

[Sean Corfield]

Jan Goyvaerts wrote:HOWEVER ... if I were to tell them Clojure will allow me to develop a certain feature much faster, I'm sure they'll let me go for it.

So, is there something that comes to mind ?

All I can say in response is that Clojure allows my team to build more complex systems faster, in much less code, and we are able to make sweeping changes more easily due to the modular, composable code that you tend to get as a result of a purely functional approach. We're also able to take advantage of multi-core machines much more easily and effectively with Clojure's safe concurrency model - immutability makes concurrency so much easier!

Some of the things we do with Clojure: interact with search engines and various JSON/XML REST services; generate and send about a million HTML emails every day; analyze multi-gigabyte logfiles; interact with MySQL and MongoDB transparently using the same document-based abstraction; and a whole bunch of "regular" programming tasks.

[Palak Mathur]

Sean Corfield wrote:

Jan Goyvaerts wrote:HOWEVER ... if I were to tell them Clojure will allow me to develop a certain feature much faster, I'm sure they'll let me go for it.

So, is there something that comes to mind ?

All I can say in response is that Clojure allows my team to build more complex systems faster, in much less code, and we are able to make sweeping changes more easily due to the modular, composable code that you tend to get as a result of a purely functional approach. We're also able to take advantage of multi-core machines much more easily and effectively with Clojure's safe concurrency model - immutability makes concurrency so much easier!

Some of the things we do with Clojure: interact with search engines and various JSON/XML REST services; generate and send about a million HTML emails every day; analyze multi-gigabyte logfiles; interact with MySQL and MongoDB transparently using the same document-based abstraction; and a whole bunch of "regular" programming tasks.

Hi Sean,

Your bumber sticker says that you have experience (or interest) in Scala as well. If you were given a choice between Clojure and Scala, which one will you select and why?

[Sean Corfield]

Palak Mathur wrote:For that matter, if they switch to a new language, then they also have to change the way their products interact with Java or go for a different product or develop such a solution on their own.

Clojure can use any existing Java libraries and systems. Clojure can also generate standard Java class APIs that can be called from existing Java code or products. Interoperability isn't an issue with Clojure.

One of the first things we wrote in Clojure was a log4j appender so that we could do more sophisticated logging. Generating a log4j-compatible class was very straightforward - just adding a :gen-class annotation to the Clojure namespace declaration and an :aot annotation to the project build file, then lein install compiles Clojure to .class files, builds a JAR and installs it in the local Maven repository.

[Sean Corfield]

Palak Mathur wrote:Your bumber sticker says that you have experience (or interest) in Scala as well. If you were given a choice between Clojure and Scala, which one will you select and why?

As noted in another thread, I've taken both Scala and Clojure to production and I much prefer Clojure. You can read why in this "Questions on Clojure" thread.

[Chas Emerick]

Sean has already hit many of the key points that I would have made in his absence, so I won't belabor them.

If the organization in question is highly conservative, and "enterprisey" in nature, with a "why should I bother?" attitude among management, then making headway with anything other than the next revision of Java is not likely to succeed. There is certainly a time and place for a conservative posture, but the "why bother?" sentiment would be particularly troublesome to me if I were in that organization. One bothers because better raw materials leads to a better outcome. If the organization is only after "good enough" (a perfectly rational approach, if software is not a driving factor in the organization's success), then those that care about improving outcomes are going to find their options hemmed in on all sides (Clojure and otherwise).

I've brought Scala applications to production as well, though this was circa 2007-2008, so Scala was much rougher around the edges back then. As Sean said, that people can, to some rough approximation, write Java in Scala is hardly indicative of Scala's ease of adoption. It is true that some may find the syntactic similarities between Java and Scala compelling; however, I don't think it's reasonable to say that Scala is rooted in Java by any stretch. It is a language with an incredibly sophisticated static type system that largely bears no relation to Java's. That's not to ding Scala — if you want its cut at static typing, and you need to be on the JVM, then it's the best game in town — but I have yet to hear of someone that's become proficient with Scala without going through the process of…becoming proficient in Scala. ;-) That is, whatever new language you adopt, you need to learn it, and knowing Java-the-language isn't going to help much at all. (Knowing how the JVM works, however, is quite helpful in either case.)

Re: the Guardian's presentation about their Scala transition, much of what they say about Scala holds true for Clojure: the reuse of the runtime, bytecode, deployment methods, tools, and so on is just as true, as is the fact that you can have mixed Java/Clojure projects without a hitch. IMO, this makes the main points of contrast between Clojure and Scala:

* dynamic vs. static typing
* s-expressions (i.e. parens) vs. braces and parens and spaces and operator associativity
* homoiconicity vs. …no homoiconicity (although Scala has some experimental stabs ongoing on a macro system of its own)
* miscellaneous stuffs, e.g. the details of how the REPL is used (absolutely critical in Clojure vs. a more auxiliary role in Scala)

My personal experience with transitioning to Clojure is after using Java exclusively for ~8 years, with some Python and Scala mixed in at the end. PDFTextStream was (and still is) written entirely in Java, but I needed something more productive to write higher-level services in, and Clojure ended up being that more productive option. Using it, dozens of large document automation systems have been built that use PDFTextStream in conjunction with dozens (hundreds?) of other Java and Clojure libraries. This endeavor has never involved a large headcount, which is what I assume is meant by "large scale environment"; the truth is that a quality Clojure programmer can run circles around an equivalently-skilled Java programmer, just as a Java programmer can run circles around a C++ programmer when building applications because of advances like garbage collection and standard collections libraries.

Chapter 19 in the book is officially titled 'Introducing Clojure to your workplace', but its secondary title is 'Sneaking Clojure Past the Boss'. I think I'll see about putting that up as a blog post or something, as questions about how to effectively bring Clojure into one's organization is a very common question.

[chris webster]

@Chas and Sean: Many thanks for the detailed comments - lots of food for thought there. You're both making a strong case for "sneaking Clojure past the boss"!

Although in a lot of places I've worked, it can be hard enough to sneak "making best use of current tools" past the boss...

[Jan Goyvaerts]

Are there engineering applications of Clojure ? My company works about optimization in engineering with lots of maths involved.

[Chas Emerick]

Jan Goyvaerts wrote:Are there engineering applications of Clojure ? My company works about optimization in engineering with lots of maths involved.

Indeed; check out Incanter, "a Clojure-based, R-like platform for statistical computing and graphics". There's scads of people using Incanter for absurdly interesting things; though, as you can imagine, most don't make much noise about it, given the industries they work in (finance, engineering, etc.).

There's also Clojuratica, a Clojure front-end to Mathematica, although it doesn't look like it's been maintained as of late.

--
(coauthor of Clojure Programming from O'Reilly; creator of Clojure Atlas)

[Jan Goyvaerts]

AHA ! NOW we're getting somewhere. :-)

Does that imply Clojure is more adequate for maths and statistics ? Compared to Java that is. Because *that* is something I can sell to IT managers.

In my branch Fortran is still much appreciated by the academics who're writing algorithms. Would it beat Fortran on the JVM for example ?

[Chas Emerick]

Jan Goyvaerts wrote:AHA ! NOW we're getting somewhere. :-)

Does that imply Clojure is more adequate for maths and statistics ? Compared to Java that is. Because *that* is something I can sell to IT managers.

In my branch Fortran is still much appreciated by the academics who're writing algorithms. Would it beat Fortran on the JVM for example ?

Is there an efficient Fortran on the JVM? I know of Fortress, but that's obviously only inspired by Fortran.

Yes, anyone that often reaches for Matlab or Mathematica or R to do statistical/maths stuff, but would like to be in a JVM environment, will likely find the Clojure/Incanter combo to be pretty compelling. Using Java for anything that these tools are used for would be straight up insane. (Not that it hasn't been done.)

However, are there any domains where Java is "more adequate" than essentially any other widely-used JVM language? Clojure will outpace Java in maths and stats work, web development, thick client dev (using either Swing via seesaw or JavaFx, infrastructure automation, "big data" stuff (e.g. driving hadoop with cascalog), and so on.

(FYI, I run a yearly 'State of Clojure' survey every year; some here — and some of your bosses ;-) — may find last year's results interesting.)

Anyway, this is the point I was trying to make earlier; any hesitation around using Clojure or Scala or JRuby, et al. is related to social and organizational factors, not whether these languages are worlds better than Java or not. For my part, I'll use any of them instead of Java; I know that my business will outrun any competitors and not have to absorb any opportunity costs due to my tech stack.

--
(coauthor of Clojure Programming from O'Reilly; creator of Clojure Atlas)

[Jan Goyvaerts]

Interesting, this State of Clojure. It's something to show to IT managers. Thanks ! I wonder if these exist for Scala also. It's something you might do for any language for that matter.

The development environments of choice are - for me - unusual. A vast majority of character-based ide. To the risk of offending certain people here, don't these tools limit the size of the projects ? (I guess you can tell I never used emacs :-).

[Sean Corfield]

Jan Goyvaerts wrote:The development environments of choice are - for me - unusual. A vast majority of character-based ide. To the risk of offending certain people here, don't these tools limit the size of the projects ? (I guess you can tell I never used emacs :-).

I'm curious as to why you think it might limit the size of projects? (and bear in mind that Clojure code tends to be an order of magnitude smaller than the equivalent Java code)

[chris webster]

Jan - you might be interested in this video where Neal Ford talks about how new technologies become (or can be made) acceptable: Master Plan for Clojure Enterprise Mindshare Domination

[Jan Goyvaerts]

@chris - thanks ! I'll have a look to it.

@Sean - Again, I intend no offence. I never used emacs. Neither am I thinking bad about it. When I'm talking about "limited" I mean projects with hundreds of classes. An ide and a high-resolution monitor are a necessity in that case; making sure nothing was broken. Something I/me/personally think as a tall order for a character-based ide. Simply because there's just so much data you can put on the screen. Nothing else. I have fond memories about the Turbo Pascal IDE but it wouldn't be able to cope with today's stuff. :-)

[Sean Corfield]

Jan Goyvaerts wrote:Again, I intend no offence.

No offense taken at all - I was just curious what you thought the limitations might be.

When I'm talking about "limited" I mean projects with hundreds of classes. An ide and a high-resolution monitor are a necessity in that case; making sure nothing was broken.

I run Emacs full-screen on a 27" iMac with 2560x1440 resolution. My workspace is typically divided into four panels but I'll often work with up to eight panels. I normally have at least one shell panel active in Emacs, continuously running tests (so every time I change either code or tests, it runs the unit test suite - which takes about 4-5 seconds - so I can see breakages immediately and maintain a TDD workflow pretty effectively. I can also run tests for a particularly file with just a keystroke if needs be. I tend to have another shell active for builds or any other commands I need to run, so I don't have to have switch out of Emacs to do that. I've always got a REPL panel active in Emacs and tend to cycle between production code, test code and the REPL as I'm developing. With Swank/Slime, I'm able to pull up documentation, jump to definitions or call sites with just a few keystrokes. I also tend to have an IRC panel active, connected to #clojure on freenode so I can ask questions about coding techniques when necessary

I actually find I'm more productive with Clojure in Emacs than I was with Clojure (or any other language) in Eclipse - but part of that might be due to my improved Clojure skills since I last used Eclipse.

There's certainly a steep learning curve with Emacs but I don't find anything missing that I had in Eclipse (except global find'n'replace feels is a bit clunky in Emacs - but then it's also pretty rare that I really need that).