ScaLearning 7 – Distributing Concurrent Tests

facebooktwitterdiggdzonestumbleuponredditdelicious


Like many developers who make the journey from Java to Scala, I often find myself amazed at how much easier it is to do some things, or how much easier it is to express myself in Scala.

“ScaLearning” will be a series of short blog-posts just documenting little tidbits I find interesting, confusing, amusing, or otherwise worthy of talking about.
 
 

Motivation

Recently in order to gain confidence in our web application, our team decided it pragmatic to run a series of tests over a deployed version of the application complete with production-like database. A simple suite of non-destructive tests we could run in any environment was quickly put together.
 
Unfortunately one of our simplest tests quickly began causing us trouble. The test emulated a search spider, crawling every link it found on the entire site in an exhaustive graph traversal complete with cycle-detection. Unfortunately, the test ran for over 24 hours without completing.
 
While we’ve made other optimizations to improve performance of the test (such as excluding sufficiently similar pages), the topic of today’s post is the concurrency we introduced in order to help take the edge off test time.
 
 

Distributing Concurrent Tests

Our goal was relatively simple. We wished to run a very simple test across many thousands of URLs:

def testSingleUrl(nextUrl : String) = {
      client.open(nextUrl)
      client.statusCode should equal(200)
}

 
 
If either of the calls inside this test fail, an exception would be thrown. This was an acceptable way of detecting test failure. However, the test needed to do a bit more – traverse the graph:

  def testUrl(nextUrl: String): Unit = {
      if (wasVisited(nextUrl)) return

      val client = new HtmlUnitRunner
      client.open(nextUrl)
      client.statusCode should equal(200)

      registerVisitedUrl(nextUrl)
      
      JListWrapper(client.currentPage.getAnchors())
              .map(_.getAttribute("href"))
              .filter(isLinkValid(_))
              .foreach((a: String) => {
                      markForTesting(nextUrl)
              })
  }

Of course, this was called by a method which pushed and popped from a stack, and “markForTesting” pushed a new link onto that stack. This code worked great sequentially, but we wanted it to operate concurrently in order to minimize testing time. For this, we employed Akka’s actors:

class ConcurrentTest extends Actor {
  def receive = {
    case name: (() => Unit) =>
      try {
        name()
        self reply(true)
      } catch {
        case t: Throwable =>
          t.printStackTrace()
          self reply(false)
      }
    case _ =>
      println("Actor received an unrecognized message")
      self reply(false)
  }
}

ConcurrentTest, as you can see, is the driver behind an individual run of the test method:

val testRunner = actorOf[ConcurrentTest].start()
val result = testRunner !!! (() => testUrl("/index.html")
// Other code can go here
result.get

 
 
Assuming the methods called within testUrl are thread-safe (which we also did using Actors), this will run a single test using a second Thread, and allow us to continue on with our business. However, since there’s only a single Actor here, we only have one Thread with which to process URLs. This means that we’re still effectively opening each link sequentially.

We need a Thread pool, and Akka is glad to provide one:

class Master(nofWorkers: Int) extends Actor {
  val workers = List.fill(nofWorkers)(actorOf[ConcurrentTest].start())
  val router = Routing.loadBalancerActor(new CyclicIterator(workers)).start()
  val answers = ListBuffer[Future[Boolean]]()

  def receive = {
    case name: (() => Unit) => answers += router !!! name
    case "result" => {
      Futures.awaitAll(answers.toList)
      self reply(answers.foldLeft(true)(_ && _.result.get))
      workers.foreach(_.stop())
      router.stop()
      self stop()
    }
  }
}

 
 
So far everything I’ve presented as code came very naturally. In fact, minor modifications for the purpose of blogging notwithstanding, we used the code I’ve presented so far to test several links across our site very successfully, in a fashion very similar to the following:

val masterRunner = actorOf(new Master(concurrentActors)).start()
val productList = fetchAllProductsFromDatabase

productList.map(productToUrl(_))
        .foreach(url => masterRunner ! (() => testSingleUrl(url)))

((masterRunner !! "result").getOrElse(false).asInstanceOf[Boolean]) should equal(true)

However, when we attempted to apply the same methodology to our crawl test, things didn’t work as well as we’d hoped:

    val masterRunner = actorOf(new Master(concurrentActors)).start()

    // def testUrl as seen above

    def markForTesting(nextUrl : String) {
            masterRunner ! (() => testUrl(nextUrl))
    }

   def fullTest() = {
            val baseUrl = "/index.html"
            masterRunner ! testUrl("baseUrl")

             ((masterRunner !! "result").getOrElse(false).asInstanceOf[Boolean]) should equal(true)
    }

 
 
The theory was that “result” would wait for all of the answers to come back before returning. Unfortunately, that’s not quite the sequence of events the Actor sees. In reality, after digging, we figured out what messages the actor received:

  1. (() => testUrl(“/index.html”)) occurs, which is quickly sent to a ConcurrentTest runner
  2. “result” comes next, as it takes a second or two for the runner to open the test
  3. (() => testUrl(_)) is received for several other URLs as links are scraped off the first page

“result” doesn’t actually wait for all the answers to come back, as it has no way of knowing how many answers are actually required. For that matter, we aren’t sure of that number either, as the test is meant to be dynamic. Instead, “result” simply compiles the answers it has so far, and then shuts down all of the actors. This means we get a “yes” or “no” about “/index.html”, but all of the other URLs are still sitting in the mailbox of Master when it’s shut down. Uh-oh!

So how do we know when we’re done? Mailbox sizes. We added a new match case to Master which would calculate if it believed the tests to be done yet:

class Master(nofWorkers: Int) extends Actor {
  ...
  def receive = {
    ...
    case "done" => { // Added this case, the rest of "Master" remained unchanged
        Futures.awaitAll(answers.toList)
        val overallSize = workers.map(worker => worker.mailboxSize).foldLeft(0)(_ + _) + router.mailboxSize
        self.reply(overallSize == 0 && self.mailboxSize == 0)
    }
  }
}

 
 
This code is different than “result” in that it actually attempts to detect if the tests are done by:

  1. Waiting for all currently outstanding test-methods to complete
  2. Counting any pending messages in the router and worker mailboxes.
    (This should always be zero, as we’ve waited for all answers to return, but it’s still safer to be sure)
  3. Counting any pending messages on the master
  4. Return “true” if the total pending messages is 0, otherwise “false” as more tests have to run

This algorithm will work for us because when we run Futures.awaitAll it runs every outstanding test to completion. Any URLs found on the pages to be tested are checked against previously-visited URLs, and added to Master’s queue if they are new. Since Master is still processing “done” those tests will stay on the queue and “mailboxSize” will return a positive non-zero number. If, however, no new links are encountered, then there will be no tests waiting on the Master queue, and our “done” operation will detect 0 pending tests.

In use:

val masterRunner = actorOf(new Master(concurrentActors)).start()
def markForTesting(nextUrl : String) = masterRunner ! (() => testUrl(nextUrl))
// def testUrl as seen above

def fullTest() = {
    val baseUrl = "/index.html"
    masterRunner ! testUrl("baseUrl")
    while((masterRunner !! "done").getOrElse(false).asInstanceOf[Boolean] == false) {
        Thread.sleep(1000);
    }
    ((masterRunner !! "result").getOrElse(false).asInstanceOf[Boolean]) should equal(true)
}

Now we sleep our thread, asking the master if it’s completed its job once every cycle, until the master claims all of its workers have completed their work and no new work is pending for the master to distribute.

Feedback as to other potential approaches is very welcome, I find the entire topic of concurrency and job distribution very interesting.


facebooktwitterdiggdzonestumbleuponredditdelicious

ScaLearning 6 – DSL Exploration & Operator Interpretation

facebooktwitterdiggdzonestumbleuponredditdelicious


Like many developers who make the journey from Java to Scala, I often find myself amazed at how much easier it is to do some things, or how much easier it is to express myself in Scala.

“ScaLearning” will be a series of short blog-posts just documenting little tidbits I find interesting, confusing, amusing, or otherwise worthy of talking about.
 
 

DSL Exploration: Our Tale So Far

For those interested in the whole DSL Exploration “mini-series”, we began with DSL Exploration & Operator Notation.

To quickly summarize our tale so far, I decided to explore the creation of a DSL to expose some of Scala’s nuances. My project of choice was codifying the LOLCode language as a Scala DSL. So far we have created a syntax for outputting data to the screen.

Working syntax so far:

U SEEZ "Hello World" !
U SEEZ "Hello World" !!

 
 

Learned Lesson 3: Operation Interpretation

So far we’ve been doing all of our work in the Scala REPL. While extremely powerful, the REPL does have a few known limitations – it sometimes interprets a line differently than it would have been interpreted from a source file. Today we’ll discover one such situation, and explore the reason.

Let us start with a Scala source-file which will act as our basis:

class Printable(output:String) {
  var bang = false;
  def ! = { print(output); }
  def !! = { println(output); }
}
class OutputStream {
  def SEEZ(output: Printable) : Unit = println(output.toString)
}
val U = new OutputStream

U SEEZ "Hello"!
U SEEZ " World"!!
U SEEZ "Goodbye"!
U SEEZ " World"!!

The code above is equivalent to what we were writing into the REPL in previous posts, but let’s store that in a file ScaLol.scala and run it:

tcalder:bin tcalder $ scala ScaLol.scala
/.../ScaLol.scala:10: error: value ! is not a member of java.lang.String
U SEEZ "Hello"!

Clearly not what the REPL was telling us. At first glance, it now appears as if it’s completely changed its operational precedence, and that’s true… but only sort of.

After trying various combinations of the code (none of which helped), I eventually gave up and started reading the Scala Reference again.

Let’s review what we’ve said so far:

  • Methods with one argument can be used as an infix operator
  • Methods with no arguments can be used as a postfix operator
  • Unary operators for !, +, -, and ~ can be used if equivalent “unary_*” methods are defined
  • Postfix operators always have lower precedence than infix operators
  • Infix operators have an order of precedence which includes letters being lower than special characters

Reasoning briefly about our code, the REPL clearly evaluates the “!” as a postfix operator, which gives it the lowest precedence and results in:

U.SEEZ("Hello").!()

Yet in ScaLol.scala we appear to have attempted to run “!” on String. Since Postfix have low precedence, we just be attempting to evaluate “!” as an Infix operator. This would make sense since special characters have a higher operational precedence:

U.SEEZ( "Hello World".!(.....?) )

Now if you’re like me, you’re wondering what on earth it could possibly be trying to fill in as the argument to this strange new Infix “!”. The answer is, it’s trying to use the next line as a continuation on the current:

U.SEEZ("Hello".!(U)).SEEZ("World"... // Or some rough equivalent

From the ScalaReference:

Scala is a line-oriented language where statements may be terminated by semicolons or newlines. A newline in a Scala source text is treated as the special token
“nl” if the three following criteria are satisfied:

  1. The token immediately preceding the newline can terminate a statement.
  2. The token immediately following the newline can begin a statement.
  3. The token appears in a region where newlines are enabled.

“U” can start a statement, and “!” can terminate one, since the REPL clearly didn’t get upset at either of these. Newlines are enabled in our code – there’s a lot of logic in the Reference that explains why if you’d like to explore it.

Based on this, we can conclude that our “\n” after the “!” is, in fact, a “[nl]”.

The second piece of the puzzle that we need is the following (summarized) syntactic definitions, expressed in the Scala Ref using Extended Backus–Naur Form:

PostfixExpr ::= InfixExpr [id [nl]]
InfixExpr ::= PrefixExpr
                     | InfixExpr id [nl] InfixExpr
PrefixExpr ::= [‘-’ | ‘+’ | ‘~’ | ‘!’] SimpleExpr

In specific, draw your attention to the similarity between the definition of “PostfixExpr” and the second definition of “InfixExpr”, and pay special attention to our friend “nl”.

The following is a (rough) breakdown of what we’ve actually attempted:

U SEEZ "Hello World" !
U SEEZ "Goodbye World" !

SimpleExpr id SimpleExpr id nl SimpleExpr id SimpleExpr id

PrefixExpr id PrefixExpr id nl PrefixExpr id PrefixExpr id

InfixExpr id InfixExpr id nl InfixExpr id InfixExpr id

Bingo, there’s our culprit. The second “id” in the expression is our “!” operator. Based on the formal specification, “InfixExpr id nl InfixExpr” can be simplified as a single “InfixExpr”, and since “!” has higher precedence than “SEEZ”, it will clearly be evaluated first.

As a final proof, try saving the following code as “ScaLol2.scala” and running it. Note how we use two new-lines in-between “SEEZ” calls. This results in two “nl” entities instead of one, which prevents the compiler from interpreting an InfixExpr where we didn’t intend one:

class Printable(output:String) {
  def ! = { print(output); }
  def !! = { println(output); }
}
class OutputStream {
  def SEEZ(output: String) : Printable = new Printable(output)
}
val U = new OutputStream

U SEEZ "Hello"!

U SEEZ " World"!!

U SEEZ "Goodbye"!

U SEEZ " World"!!

The result on my machine:

tcalder:bin tcalder $ scala ScaLol2.scala
Hello World
Goodbye World

That’s all for this week! I hope today’s post proved interesting, it was far deeper than I ever expected to go, which was exactly what I was hoping for.


facebooktwitterdiggdzonestumbleuponredditdelicious

ScaLearning 5 – DSL Exploration & Operation Precedence

facebookdiggdzonestumbleuponredditdelicious


Like many developers who make the journey from Java to Scala, I often find myself amazed at how much easier it is to do some things, or how much easier it is to express myself in Scala.

“ScaLearning” will be a series of short blog-posts just documenting little tidbits I find interesting, confusing, amusing, or otherwise worthy of talking about.
 
 

DSL Exploration: Our Tale So Far

For those interested in the whole DSL Exploration “mini-series”, we began with DSL Exploration & Operator Notation.

To quickly summarize our tale so far, I decided to explore the creation of a DSL to expose some of Scala’s nuances. My project of choice was codifying the LOLCode language as a Scala DSL. So far we have created a syntax for outputting data to the screen.

Working syntax so far:

U SEEZ "Hello World"

 
 

Learned Lesson 2: Operation Precedence

Now that we’ve successfully defined “U SEEZ”, the LOLCode spec allows us to add a bang at the end of a sentence to “cancel” the newline. While this feels a little backwards, we can attempt it.

Our goal:

U SEEZ "Hello World" // Includes newline
U SEEZ "Goodbye"! // No newline

My first plan of attack was to make “SEEZ” return an object that has the “!” method, which would do a “print” instead of a “println”. Fortunately, before attempting this solution I realized that if I were to do so there would be no method call to trigger the non-bang “println”.

Attempt two, maybe we can allow SEEZ to accept a different object. That object can have a “!” method. Here’s what I’m aiming for:

U.SEEZ( "Hello World".!() )

 
We’ll also need an implicit to turn Strings into this special new object. Here goes.

scala> class Printable(val text : String, val bang : Boolean) {
     | override def toString() = if (bang) {text} else {text + "\n"}
     | def ! = new Printable(text, !bang)                           
     | }                                                            
defined class Printable

scala> class OutputStream {
     | def SEEZ(p : Printable) = print(p.toString())
     | }
defined class OutputStream

scala> val U = new OutputStream
U: OutputStream = OutputStream@e208506

scala> implicit def stringToPrintable(s : String) : Printable = new Printable(s, false)
stringToPrintable: (s: String)Printable

scala> U SEEZ "Hello World"
Hello World

That’s good news, we’ve managed to maintain the old behaviour, so let’s give our new “!” method a try:

scala> U SEEZ "Hello World"!
:10: error: value ! is not a member of Unit

Foiled again! So what happened here? As “println” returns type “Unit”, which is the Scala equivalent to “void”, we can venture a guess that the “!” method was actually run on the result of “println”:

U.SEEZ("Hello World").!()

 
As I’m sure you’ve guessed, the issue at hand is operator precedence. Let’s experiment a little:

scala> class MyObj(val v : String) {                                                  
     | def in(m : MyObj) = {println("[" + v + ".in(" + m.v + ")]"); new MyObj(v + m.v) }
     | def post() = {println("[" + v + ".post()]"); this}
     | def unary_+ = {println("[" + v + ".pre()]"); this}
     | } 
defined class MyObj
   
scala> implicit def stringToMyObj(s : String) : MyObj = new MyObj(s);
stringToMyObj: (s: String)MyObj

scala> +"a"
[a.pre()]
res2: MyObj = MyObj@7987b796

scala> + "a" in "b"
[a.pre()]
[a.in(b)]
res3: MyObj = MyObj@3cd41115

scala> + "a" in "b" post
[a.pre()]
[a.in(b)]
[ab.post()]
res4: MyObj = MyObj@6dcc55fb

scala> "a" in "b" in "c"
[a.in(b)]
[ab.in(c)]
res5: MyObj = MyObj@69cad977

scala> "a" in "b" post
[a.in(b)]
[ab.post()]
res6: MyObj = MyObj@37b24706

scala> "a" post
[a.post()]
res7: MyObj = MyObj@7ea4b9da

scala> "a".post in "b"
[a.post()]
[a.in(b)]
res8: MyObj = MyObj@7e28388b

scala> "a" post in "b"
:1: error: ';' expected but string literal found.
       "a" post in "b"
                   ^

 
 
We seem to be allowed to omit all the dots and brackets when we chain infix operators, but we’re unable to chain an infix operator to the result of a postfix operator unless the postfix used a dot to call. The postfix call at the end of a chain also seems to apply to the result of the previous infix call.

Note: By using the ‘dot’ in [“a”.post in “b”] we are no longer using an ‘operator’. Explicit method calls are referred to as “function applications” in the reference

According to the Scala Reference:

  • If there are several infix operations, operators with higher precedence bind more closely than operators with lower precedence
  • If there are consecutive infix operations with operators of the same precedence, all operators must have the same associativity
  • Postfix operators always have lower precedence than infix operators

While prefix operators aren’t mentioned, they appear to have a higher precedence than infix operators via experimentation:

scala> class MyObj(val v : String) {
     | def +(m : MyObj)  = {println("[" + v + ".in(" + m.v + ")]"); new MyObj(v + m.v) }
     | def unary_+ = {println("[" + v + ".pre()]"); this}
     | }
defined class MyObj
   
scala> implicit def stringToMyObj(s : String) : MyObj = new MyObj(s);
stringToMyObj: (s: String)MyObj

scala> +"a" + +"b"
[a.pre()]
[b.pre()]
[a.in(b)]
res21: MyObj = MyObj@144f1ada

The end result of this discussion is another compromise. After experimenting with various notations, I gave up and returned to my original plan of a “!” method on an object returned by “SEEZ”:

scala> class Printable(output:String) {
     | def ! = { print(output); }
     | def !! = { println(output); }
     | }
defined class Printable

scala> class OutputStream {
     | def SEEZ(output: String) : Printable = new Printable(output)
     | }
defined class Printable

scala> val U = new OutputStream
U: OutputStream = OutputStream@608d116e

scala> U SEEZ "Hello World"!
Hello World
scala> U SEEZ "Hello World"!!
Hello World

scala>

Stay tuned, next time we’ll attempt to put the pieces together and run HelloLol.scala


facebookdiggdzonestumbleuponredditdelicious

ScaLearning 4 – DSL Exploration & Operator Notation

facebooktwitterdiggdzonestumbleuponredditdelicious


Like many developers who make the journey from Java to Scala, I often find myself amazed at how much easier it is to do some things, or how much easier it is to express myself in Scala.
 
“ScaLearning” will be a series of short blog-posts just documenting little tidbits I find interesting, confusing, amusing, or otherwise worthy of talking about.
 
 

Why make a DSL?

I’ve heard it said that the real meat of learning Scala is done by people who want to expose a DSL. So far I’ve found learning Scala fairly easy. Sometimes I’m a little unclear on rules, like when I can leave out brackets and when I can’t, but when that happens I can just re-add the brackets and carry on.
 
The same flexibility isn’t quite so true for people who wish to create a good DSL. A great deal of effort goes into carefully constructing the syntax. Scala is such an incredibly extensible language that, with a little patience, you can create code that doesn’t look at all like “Scala”, and yet still compiles.
 
In order to get a taste of this learning, I decided I was going to attempt a DSL.
 
 

Starting into DSL Exploration

I decided my DSL of choice was going to be LOLCode.
 
Based on the (priceless) LolCats meme, LolCode presented itself as the perfect project to build a DSL around:

  • Some rigidity in sticking close to the community consensus on syntax
  • Enough flexibility to allow me to explore a few different approaches
  • Different enough from Scala to present a challenge in implementation
  • There’s a specific “feel” for the DSL we can aim for
  •  
     

    Learned Lesson 1: Operator Notation

    I decided to start where everyone starts, “Hello World”. System output is one of the most basic pieces of functionality for a language. The spec lists “VISIBLE” as the keyword for println:
    VISIBLE "Hello World" // Prints hello world with a newline
    VISIBLE "Goodbye"! // Prints goodbye, but with no newline

     
    This seemed easy enough. We’ll start by ignoring the bang-syntax:
    scala> def VISIBLE(text : String) = {println(text)}
    VISIBLE: (text: String)Unit
     
    scala> VISIBLE "Hello World"
    :1: error: ';' expected but string literal found.
           VISIBLE "Hello World"

     
    Uh-oh.
     
    Just starting in, we’ve already ran into a lesson on operator syntax. There are three kinds of Operator Notation:

    • Unary (prefix) Notation
      • Only works for +, -, !, and ~
      • Method must be no-args
      • Must omit the dot – ie: !.true would not work, but !true would
    • Postfix Notation
      • Method must be no-args
      • May omit the dot and brackets if desired, but not required.
    • Infix Notation
      • Method must be single-argument
      • May omit the dot and brackets if desired, but not required. ie: “a” + “b” or “a”.+(“b”)

     
    In my case, “VISIBLE” would be a good Unary operator on some string-like object, but it is not a valid unary operator. For it to be an infix operator, I need some other “object” for it to be Infix of. Looks like I’ll have to compromise on this one.
     
    Reading around the site, I discovered that while the spec listed “VISIBLE” as the “println” equivalent, “U SEEZ” was the community’s preference.
     
    U SEEZ "Hello World"
    U SEEZ "Hello World"!

     
    This is convenient, as commenters on LolCode pointed out, as “U” can be seen as the I/O similar to Java’s “System.out”, and “SEEZ” is simply a “print” method on any output stream.
     
    So let’s wrap up today’s post with one last crack at the non-bang notation:
    scala> class OutputStream {
         | def SEEZ(output: String) = println(output);
         | }
    defined class OutputStream
     
    scala> val U = new OutputStream
    U: OutputStream = OutputStream@4be03c55
     
    scala> U SEEZ "Hello World"
    Hello World

     
    Worked like a charm. Stay tuned, as I continue to explore the process behind creating a DSL, and what it reveals about the language.


    facebooktwitterdiggdzonestumbleuponredditdelicious

ScaLearning 3 – Is Scala Hard to Learn?

facebooktwitterdiggdzonestumbleuponredditdelicious


Like many developers who make the journey from Java to Scala, I often find myself amazed at how much easier it is to do some things, or how much easier it is to express myself in Scala.

“ScaLearning” will be a series of short blog-posts just documenting little tidbits I find interesting, confusing, amusing, or otherwise worthy of talking about.
 
 

Is Scala Hard to Learn?

If you want to skip the explanation, my answer is no.

To elaborate, I’ve heard a great deal of “buzz” about how complicated Scala is. I’ve heard it said that it will become an academic language, and that average programmers won’t be able to pick it up. Unfortunately, the blogosphere is so cluttered with rebuttals to the idea that it’s hard to find anyone actually critiquing Scala.

Some examples of pro-scala posts:

Is Scala Complex? (Martin Odersky)
Is Scala Too Complex? (Eric Daugherty)
Is Scala More Complicated than Java? (Vassil Dichev)

Some examples critiquing the complexity:

My Verdict on the Scala Language (Doug Pardee)
Is the Scala 2.8 Collections Library a Case of the Longest Suicide Note in History? (oxbow_lakes)

Since a lot has been said already, I won’t go into any in-depth tirade about software complexity. I will, however, draw on my experience:

For me, Scala has been easy to learn after a few years of Java because:

  • Type system that’s similar to Java
  • Java interoperability allows me to ease myself in
  • It’s possible to write near-Java syntax, and have it be valid Scala
  • Many Scala keywords have direct analogs in Java (ie: public final String s vs val s : String

 
 
So far I’ve done two “projects” in Scala.

One was a Lift (version 1.x) application that talked to a back-end set of Jersey services. The native XML handling was a dream-come-true, but the mapping of Lift 1.0 made me think Scala was much harder than it is.

The second project was an experimental conversion of a Spring MVC application using JSP views into a Lift 2.x website. I left the backing services in Java, used Spring to get the dependencies, but rewrote the front-end to be Lift. The mapping I hated in 1.x had been completely overhauled, and was actually very pleasant to work with.
 
 
That brings me to the take-away of this post.
 
 
A large part of the “complexity” people see in Scala isn’t the language. It’s the libraries and frameworks.

Scala, as a language, is very simple and very powerful. Overall, while I don’t know every feature, I feel as if I know enough to be able to understand most of the code I read, and colleagues familiar with Scala seem to be able to read the code I’m writing.

I find Scala very easy to learn. Some libraries have been a challenge… but then, so is learning the exact magic incantation for Java’s file IO.


facebooktwitterdiggdzonestumbleuponredditdelicious

ScaLearning 2 – Reducing string-join to a line

facebooktwitterdiggdzonestumbleuponredditdelicious


Like many developers who make the journey from Java to Scala, I often find myself amazed at how much easier it is to do some things, or how much easier it is to express myself in Scala.

“ScaLearning” will be a series of short blog-posts just documenting little tidbits I find interesting, confusing, amusing, or otherwise worthy of talking about.

String Joining

I recently found myself writing a SQL generator that took a domain-specific query object and turned it into raw SQL. We had to write our own, as we needed to optimize several parts of the query, or face the rage of the customer when things were a little sluggish.

(It’s generally bad for business when the customer goes Hulk on you.)

As happens often in generating SQL, I found myself wanting to store a list and later convert it into a comma-separated string.

Java – First Attempt

StringBuilder result = new StringBuilder();
List items = Arrays.asList("a", "b", "c", "d");
boolean first = true;
for(String i : items) {
    if(first) first = false;
    else result.append(", ");
 
    result.append(i);
}
System.out.println(result.toString());

Sloppy. Three lines in the for loop, a local variable hanging around. It’s readable, but having to read 5-7 lines just to realize it’s a string-join seems excessive. Try again!

Java – Second Attempt

StringBuilder result = new StringBuilder();
List items = Arrays.asList("a", "b", "c", "d");
for(String i : items) {
    result.append(", ").append(i);
}
System.out.println(result.toString().replaceFirst(", ", ""));

Shorter, no local variable, but now we’re doing “unwork” to get rid of the leading comma. That’s an easy detail to miss, so fewer lines but not much more readable.

(I’m open to cleaner implementations! Comment with what you prefer.)

Scala

In Scala (and many other languages, ie: PHP) this is a simple one-liner.


val items = List("a", "b", "c", "d");
println( items.reduceLeft(_ + ", " + _) )
println( items.mkString(", ") )

Two ways, both one line. Both are readable, and I don’t have a real preference between them.

While readability and line count aren’t always so tied, the fewer lines a person has to read the faster they can read them. While I could easily write a method for Java to join Strings, this is just another example of how Scala seems to actively try to make my life easier.


facebooktwitterdiggdzonestumbleuponredditdelicious

ScaLearning 1 – Closure Oddities

facebooktwitterdiggdzonestumbleuponredditdelicious


Welcome to my first ScaLearning post.

Like many developers who make the journey from Java to Scala, I often find myself amazed at how much easier it is to do some things, or how much easier it is to express myself in Scala.

“ScaLearning” will be a series of short blog-posts just documenting little tidbits I find interesting, confusing, amusing, or otherwise worthy of talking about.
 
 

ScalaWTF.txt

Back in March I created a document called “ScalaWTF.txt”. Today I randomly found it, read it, and thought to myself, “WTF?! How does that work!”. Yes, I think in ancronyms 🙂

Here’s what the document said:

scala> (1 to 5) filter{x => println("hi"); x%2==0}                          
hi
hi
hi
hi
hi
res61: scala.collection.immutable.IndexedSeq[Int] = Vector(2, 4)

scala> (1 to 5) filter{println("hi"); _%2==0}                               
hi
res62: scala.collection.immutable.IndexedSeq[Int] = Vector(2, 4)

It only took me a moment or two to remember why I found this so confusing. The two closures I supply to “filter” look nearly identical, yet they behave very differently.

I decided to dig a little deeper:

scala> val f : (Int) => Boolean = (x) => {println("hi"); x % 2 == 0 }
f: (Int) => Boolean = 

scala> f(2)                                                          
hi
res1: Boolean = true

scala> f(3)
hi
res2: Boolean = false

Interesting, so that behaves as I would expect. The “println”, being a part of the closure, is executed on each call to “f”. Now the other:

scala> val f : (Int) => Boolean = {println("hi"); _ % 2 == 0 }    
hi
f: (Int) => Boolean = 

scala> f(3)                                                   
res3: Boolean = false

scala> f(2)
res4: Boolean = true

Strange, and not what I expected. Let’s take it a step further.

scala> val f : (Int) => Boolean = {println(_); 2 % 2 == 0 }    
:5: error: missing parameter type for expanded function ((x$1) => println(x$1))
       val f : (Int) => Boolean = {println(_); 2 % 2 == 0 }
                                           ^
:5: error: type mismatch;
 found   : Boolean(true)
 required: (Int) => Boolean
       val f : (Int) => Boolean = {println(_); 2 % 2 == 0 }

So apparently, while I assumed what I was doing was passing in a closure to “f”, what I was actually doing was evaluating a block, the last element of which must be a closure. Also, only the last statement is capable of binding with the wildcard.

Based on that, the following will not work:

val f : (Int,Int) => Boolean = {println(_); _ % 2 == 0 }

But this one should:

val f : (Int,Int) => Boolean = {println("a"); _ % _ == 0 }

After trying it, the REPL proved my theory. The take-home is that using wildcards in your closure isn’t quite the same as explicitly declaring your variables.


facebooktwitterdiggdzonestumbleuponredditdelicious