Taking the Next Step

[Dan King]

At this point, I'm comfortable with most Clojure concepts and syntax. To further my understanding I want to put together code that can be used for simple (file) data analysis. However, I've been unable to find any resources that explain how to read, write, and perform regex either 'natively' or via java. Can anyone suggest resources -- i.e. books, websites, examples -- that can help? Thanks.

[Hussein Baghdadi]

I think you mean that you are comfortable with Clojure lack of syntax
Clojure FAQ page contains a couple of books but you can have a taste of RegEx in Clojure right now:
http://clojure.org/other_functions

On general those functions are what Clojure has for RegEx:
#"pattern" re-pattern re-matcher re-find re-matches re-groups re-seq

If what is blocking you now is how to do RegExp in Clojure, then don't get any book, just check the previous functions.
Hope this helps.

[Sean Corfield]

Dan,

If you can provide a bit more background on what you're thinking of trying, we can give you better pointers. Is it just RegEx? If so, John's answer should help. Is is also how to read lines from a file? Jump on clojuredocs.org and search for reader, with-open, line-seq and see if that helps.

For books: Clojure Programming (O'Reilly Rough Cuts) has a lot of practical examples, as does Clojure in Action (Manning). Joy of Clojure (Manning) has some specific file reading examples but is more about the why / idiom of Clojure.

HTH,
Sean

[Dan King]

@John: Thanks for the link on regex and the list of regex functions; the information was very helpful.

Sean Corfield wrote:Dan,
If you can provide a bit more background on what you're thinking of trying, we can give you better pointers. Is it just RegEx? If so, John's answer should help. Is is also how to read lines from a file? Jump on clojuredocs.org and search for reader, with-open, line-seq and see if that helps.

Firstly, thanks for the link to clojuredocs.org and the 'reader', 'with-open', 'line-seq' search terms.

No, it's not just RegEx, but rather reading from a file, regex, and writing to a file.

I think it's best, as you suggested, for me to elaborate on what I'm trying to do.

I have a log file, in xml format, that I want to parse for 'author' data. I want to then write this parsed data to another file. For example, assuming the following xml log file:

<entries>
 <entry>
  <author>dking</author>
   ...
 </entry>
 <entry>
  <author>jdoe</author>
  ...
 </entry>
</entries>

I want to generate the following file (the email domain is the same for all authors):

dking = [e-mail]
jdoe = [e-mail]

Thanks for the help.

[Hussein Baghdadi]

I learnt to break my code into functions when coding in Clojure, after all functions are the building blocks in Clojure.
That is true for many language but it is extremely true for languages like Clojure and Haskell.
It is tempting to code the whole logic in one function like the imperative style, but I think Clojure will try to block you if you tried to do so.

[Hussein Baghdadi]

Have a look at this thread: https://coderanch.com/t/544295/clojure/Incubator/Analyzing-massive-data-Clojure
It might be useful.

[Dan King]

John Todd wrote:Have a look at this thread: https://coderanch.com/t/544295/clojure/Incubator/Analyzing-massive-data-Clojure
It might be useful.

Thanks for the link. There's a quite a lot to absorb; certainly seems useful.

As an aside, below is what I hacked together to solve the problem:

(with-open [rdr (clojure.java.io/reader "./log.xml")]
    (with-open [wtr (java.io.BufferedWriter. (java.io.FileWriter. "log_copy.txt"))]
        (doseq [line (line-seq rdr)] 
            (if (re-matches #"<author>(.*)</author>" line) 
                (.write wtr line)))))

[Sean Corfield]

Since Clojure has some powerful functions to process XML, you might want to try something like this:

(let [authors (map (comp first :content) (filter (fn [node] (= (:tag node) :author)) (xml-seq (clojure.xml/parse "./log.xml"))))]
  (spit "log_copy.txt" (apply str (interpose "\n" (map (fn [author] (str author " = " author "@emailaddress.com")) authors)))))

This probably looks a lot more complicated but...

The first line parses the XML and turns it into a walked sequence of nodes, then filters out just those nodes that have the tag author, then the map returns the first element of the content (the author name). So authors is the sequence ("dking" "jdoe").

The second line maps each author name to the string author = [e-mail], then interposes a newline string between each element, and finally apply str (sequence of strings) turns it into a single string which spit then writes to a file.

I'd definitely break that up into multiple functions so that they're easier to reuse but it should give you an idea of the power of sequences.

Here's the code broken up some more:

(defn to-xml-seq [file] (xml-seq (clojure.xml/parse file)))

(defn find-by-tag [tag xseq] (filter #(= (:tag %) tag) xseq))

(defn get-content [node] (first (:content node)))

(defn make-lines [s] (apply str (interpose "\n" s)))

(let [authors (map get-content (find-by-tag :author (to-xml-seq "./log.xml")))
      author-email #(str % " = " % "@emailaddress.com")]
  (spit "log_copy.txt" (make-lines (map author-email authors))))

[Dan King]

Sean Corfield wrote:Since Clojure has some powerful functions to process XML, you might want to try something like this...

(defn to-xml-seq [file] (xml-seq (clojure.xml/parse file)))

(defn find-by-tag [tag xseq] (filter #(= (:tag %) tag) xseq))

(defn get-content [node] (first (:content node)))

(defn make-lines [s] (apply str (interpose "\n" s)))

(let [authors (map get-content (find-by-tag :author (to-xml-seq "./log.xml")))
      author-email #(str % " = " % "@emailaddress.com")]
  (spit "log_copy.txt" (make-lines (map author-email authors))))

Thanks for the code example; it was very helpful in understanding how to parse xml data. Definitely seems clojure's sweet-spot is data analysis.

If and when you have the time, could you share an example demonstrating the creation of xml data with clojure?

Thanks.

Best,
Dan

[Sean Corfield]

Dan King wrote:If and when you have the time, could you share an example demonstrating the creation of xml data with clojure?

If you build a nested map that represents the structure of the XML, you can use (clojure.xml/emit my-map) to write the XML to *out* (stdout) and therefore wrap it in (with-out-str ...) to get back a string representation, instead of writing to *out* - or use (binding [*out* (writer "myoutput.xml")] ...)

For example:

(def authors ["sean" "dan"])

(defn make-author [author] {:tag :author :content [author]})

(defn make-entry [author] {:tag :entry :content [(make-author author)]})

(defn make-entries [authors] {:tag :entries :content (for [a authors] (make-entry a))})

(spit "myoutput.xml" (with-out-str (clojure.xml/emit (make-entries authors))))

You'll get the following in myoutput.xml:
<?xml version='1.0' encoding='UTF-8'?> <entries> <entry> <author> sean </author> </entry> <entry> <author> dan </author> </entry> </entries>

[Hussein Baghdadi]

Clojure continues to impress me.
I have a large XML file to parse and analyze and insert the result into a data base.
Yes the task isn't difficult but I'm really amazed by the elegant, succinct and dense Clojure code. Mission done by only 35 ~ 40 line of code !

[Dan King]

@Sean: Thanks for the code examples, they were very helpful.

John Todd wrote:Clojure continues to impress me.

Agreed. It's a vicious cycle: the more I learn about clojure, the more I like it...

[Hussein Baghdadi]

Please inform us once you got the symptom: "I don't want to code anything without Clojure"