'somewhat-functional-programmer' via Clojure
2018-11-14 13:31:46 UTC
This is somewhat of a retrospective -- so please bear with me. I've had the privilege of working on a clojure project for a couple of years now, and have accumulated some 15-20k lines of clojure code. I'm taking a little time to look back over what has worked for me and what hasn't in terms of code/project organization -- and *I'd love to know what has worked for other people (or hasn't)* for similarly large projects.
I knew my project was going to grow to at least as much code as it has now at the start, and my domain problem was fairly well-defined. From the very beginning I organized my code into many (15-20) different clojure 'projects' using lein. Rather than organizing code into these projects *by function area*, I found myself organizing code into projects by their *dependencies*. So any code that used libraries X,Y,Z went into a project that declared those dependencies -- even if a function made sense in a different namespace by name -- if it needed deps that I already had in another project, I moved the function to that project. For example, in the Java GIS world, if you do anything with swing components, you can easily pull in 100s of MBs of dependencies. My project involves GIS work both server-side rest-apis, but it's also nice to pull up a quick swing component showing data on a map for debugging etc. I don't have these things in the same project even though there is some library overlap because the GUI deps are just too many.
I think for me on my project, the only reason to separate anything into 'projects' is for reuse based on dependencies -- i.e., to use a function/set of functions I've written again, what's the minimal amount of deps to pull in. It's worked well for me in that sense -- I'm able to create a new 'main' project, include libraries that I want from my project, and not pull in 2,000 dependencies unless I need it. The dependencies I'm mostly concerned with here are Java deps and not clojure deps. I've found a relatively small core set of clojure deps I almost always want available to me (specter, timbre, core.async) -- though even still I have a utility library project where I have a hard rule of zero dependencies (for basic macros like (ignore-exception body-forms)).
I've used lein's checkouts and managed dependencies fairly successfully though I still forget to lein install everything before I lein uberjar my final delivery artifacts and end up debugging old code before realizing what happened (and my way of installing everything is a bash for loop :-)).
I've found this approach somewhat tedious and have been wondering if there's a better way -- and am very curious what others do.
What I've been playing around with lately is a different concept for my own code organization:
- What if all my clojure code could go in one place, or one project? (Even if it ended up being 20k+ lines of code)
- What if namespaces contained their required dependencies in their metadata?
- What if upon namespace creation, a namespace's dependencies were automatically added to the classpath?
- What if functions declared in a namespace could also declare additional dependencies? These would be added to the classpath upon first invocation of the function. This is great for my seldom used functions that need many dependencies -- code could live in the namespace that is matches its function instead of squirreled away in a project just to match its deps.
I have written a basic library that does these things, and am currently trying it out on a small scale. Concerns I've already been trying to address:
- Dynamically adding things to the classpath is generally considered a /bad/ thing to do.
- My code reads all pom.xmls on the classpath to determine what libraries are already on the classpath -- and does not re-add libraries that are already on the classpath. I think this alone takes care of a lot of issues with dynamically adding deps to the classpath (multiple versions of libraries on the classpath).
- Production deployments should not need to calculate classpaths dynamically
- This technique should be able to be used whether the classpath is precomputed (i.e., for production), or dynamically during development.
Example namespace loaded deps:
(ns test
{:dependencies '[[[[com.taoensso/timbre "4.10.0"]]]]}
(:require [taoensso.timbre :as timbre
:refer [info debug error warn spy]]))
-or-
(ns test
{:deps '{[[com.taoensso/timbre {:mvn/version "4.10.0"]]}}}
(:require [taoensso.timbre :as timbre
:refer [info debug error warn spy]]))
Now these namespace deps would be loaded dynamically by aliasing the ns macro for development with one that loads deps dynamically. In production, the metadata is simply attached to the namespace per normal use of the ns form, no deps added dynamically.
(defn-deps test-fn-deps
"Test Function with optional deps"
{:dependencies '[[diffit "1.0.0"]
[com.taoensso/timbre "4.10.0"]]
:require '([diffit.vec :as d]
[taoensso.timbre :as timbre
:refer [log trace debug info warn error fatal report
logf tracef debugf infof warnf errorf fatalf reportf
spy get-env]])
:import '((java.util HashMap)
(java.io InputStream))}
[x y]
(info :hello)
(d/diff x y))
Using defn-deps is /not/ transparent, as it does requires and imports. This I think is ok though, for a couple of reasons:
- If the deps are already on the classpath (production), no changes, and if they aren't, you get an exception (this is good -- it's an optional function that you chose not to include its deps)
- These functions should be pretty rare -- they are really functions that are useful if certain optional dependencies are on the classpath.
Thoughts?
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to ***@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+***@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
I knew my project was going to grow to at least as much code as it has now at the start, and my domain problem was fairly well-defined. From the very beginning I organized my code into many (15-20) different clojure 'projects' using lein. Rather than organizing code into these projects *by function area*, I found myself organizing code into projects by their *dependencies*. So any code that used libraries X,Y,Z went into a project that declared those dependencies -- even if a function made sense in a different namespace by name -- if it needed deps that I already had in another project, I moved the function to that project. For example, in the Java GIS world, if you do anything with swing components, you can easily pull in 100s of MBs of dependencies. My project involves GIS work both server-side rest-apis, but it's also nice to pull up a quick swing component showing data on a map for debugging etc. I don't have these things in the same project even though there is some library overlap because the GUI deps are just too many.
I think for me on my project, the only reason to separate anything into 'projects' is for reuse based on dependencies -- i.e., to use a function/set of functions I've written again, what's the minimal amount of deps to pull in. It's worked well for me in that sense -- I'm able to create a new 'main' project, include libraries that I want from my project, and not pull in 2,000 dependencies unless I need it. The dependencies I'm mostly concerned with here are Java deps and not clojure deps. I've found a relatively small core set of clojure deps I almost always want available to me (specter, timbre, core.async) -- though even still I have a utility library project where I have a hard rule of zero dependencies (for basic macros like (ignore-exception body-forms)).
I've used lein's checkouts and managed dependencies fairly successfully though I still forget to lein install everything before I lein uberjar my final delivery artifacts and end up debugging old code before realizing what happened (and my way of installing everything is a bash for loop :-)).
I've found this approach somewhat tedious and have been wondering if there's a better way -- and am very curious what others do.
What I've been playing around with lately is a different concept for my own code organization:
- What if all my clojure code could go in one place, or one project? (Even if it ended up being 20k+ lines of code)
- What if namespaces contained their required dependencies in their metadata?
- What if upon namespace creation, a namespace's dependencies were automatically added to the classpath?
- What if functions declared in a namespace could also declare additional dependencies? These would be added to the classpath upon first invocation of the function. This is great for my seldom used functions that need many dependencies -- code could live in the namespace that is matches its function instead of squirreled away in a project just to match its deps.
I have written a basic library that does these things, and am currently trying it out on a small scale. Concerns I've already been trying to address:
- Dynamically adding things to the classpath is generally considered a /bad/ thing to do.
- My code reads all pom.xmls on the classpath to determine what libraries are already on the classpath -- and does not re-add libraries that are already on the classpath. I think this alone takes care of a lot of issues with dynamically adding deps to the classpath (multiple versions of libraries on the classpath).
- Production deployments should not need to calculate classpaths dynamically
- This technique should be able to be used whether the classpath is precomputed (i.e., for production), or dynamically during development.
Example namespace loaded deps:
(ns test
{:dependencies '[[[[com.taoensso/timbre "4.10.0"]]]]}
(:require [taoensso.timbre :as timbre
:refer [info debug error warn spy]]))
-or-
(ns test
{:deps '{[[com.taoensso/timbre {:mvn/version "4.10.0"]]}}}
(:require [taoensso.timbre :as timbre
:refer [info debug error warn spy]]))
Now these namespace deps would be loaded dynamically by aliasing the ns macro for development with one that loads deps dynamically. In production, the metadata is simply attached to the namespace per normal use of the ns form, no deps added dynamically.
(defn-deps test-fn-deps
"Test Function with optional deps"
{:dependencies '[[diffit "1.0.0"]
[com.taoensso/timbre "4.10.0"]]
:require '([diffit.vec :as d]
[taoensso.timbre :as timbre
:refer [log trace debug info warn error fatal report
logf tracef debugf infof warnf errorf fatalf reportf
spy get-env]])
:import '((java.util HashMap)
(java.io InputStream))}
[x y]
(info :hello)
(d/diff x y))
Using defn-deps is /not/ transparent, as it does requires and imports. This I think is ok though, for a couple of reasons:
- If the deps are already on the classpath (production), no changes, and if they aren't, you get an exception (this is good -- it's an optional function that you chose not to include its deps)
- These functions should be pretty rare -- they are really functions that are useful if certain optional dependencies are on the classpath.
Thoughts?
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to ***@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+***@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.