alex/goldsmith
1
Fork 0
Code Releases Activity

Merge branch 'master' into dev

Browse Source
This commit is contained in:
Alex Yatskov 2016-06-06 22:52:07 -07:00
commit 87e5073c63
8 changed files with 429 additions and 272 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

122
README.md
View File

@ -1,14 +1,124 @@
# Goldsmith # # Goldsmith #
Goldsmith is a static website generator developed in Go with flexibility, extensibility, and performance as primary Goldsmith is a static website generator developed in Golang with flexibility, extensibility, and performance as primary
design considerations. With Goldsmith you can easily build and deploy any type of site, whether it is a personal blog, design considerations. With Goldsmith you can easily build and deploy any type of site, whether it is a personal blog,
image gallery, or a corporate homepage; the tool no assumptions are made about your layout or file structure. Goldsmith image gallery, or a corporate homepage; the tool no assumptions are made about your layout or file structure. Goldsmith
is trivially extensible via a plugin architecture which makes it simple to perform complex data transformations is trivially extensible via a plugin architecture which makes it simple to perform complex data transformations
concurrently. concurrently. A growing set of core plugins, [Goldsmith-Plugins](../goldsmith-plugins/), is provided to make it easier
to get started with this tool to generate static websites.
## Motivation ## ## Motivation ##
Why in the world did I make yet another static site generator? At first, I didn't think I needed to; after all, there is Why in the world would one create yet another static site generator? At first, I didn't think I needed to; after all,
a wide variety of open source tools freely available for use. Surely one of these applications would allow me to build there is a wide variety of open source tools freely available for use. Surely one of these applications would allow me
my portfolio page exactly the way I want right? After trying several static generators, namely Pelican, Hexo, Hugo, and to build my portfolio page exactly the way I want right?
Metalsmith, I found that although sometimes coming close, no tool gave me exactly what I wanted.
After trying several static generators, namely [Pelican](http://blog.getpelican.com/), [Hexo](https://hexo.io/), and
[Hugo](https://gohugo.io/), I found that although sometimes coming close, no tool gave me exactly what I needed.
Although I hold the authors of these applications in high regard and sincerely appreciate their contribution to the open
source community, everyone seemed overly eager to make assumptions about content organization and presentation.
Many of the static generators I've used feature extensive configuration files to support customization. Nevertheless, I
was disappointed to discovered that even though I could approach my planned design, I could never realize it. There was
always some architectural limitation preventing me from doing something with my site which seemed like basic
functionality.
* Blog posts can be tagged, but static pages cannot.
* Image files cannot be stored next to content files.
* Navbar item activated when viewing blog, but not static pages.
* Auto-generated pages behave differently from normal ones.
Upon asking on community forms, I learned that most users were content to live with such design decisions, with some
offering workarounds that would get me halfway to where I wanted to go. As I am not one to make compromises, I kept
hopping from one static site generator to another, until I discovered [Metalsmith](http://www.metalsmith.io/). Finally,
it seemed like I found a tool that gets out of my way, and lets me build my website the way I want to. After using this
tool for almost a year, I began to see its limits.
* The extension system is complicated; it's difficult to write and debug plugins.
* Quality of existing plugins varies greatly; I found many subtle issues.
* No support for parallel processing (this is a big one if you process images).
* A full [Node.js](https://nodejs.org/) is stack (including dependencies) is required to build sites.
Rather than making do with what I had indefinitely, I decided to use the knowledge I've obtained from using various
static site generators to build my own. The *Goldsmith* name is a reference to both the *Go* programming language I've
selected for this project, as well as to *Metalsmith*, my inspiration for what an static site generator could be.
The motivation behind Goldsmith can be described by the following principles:
* Keep the core small and simple.
* Enable efficient, multi-core processing.
* Add new features via user plugins.
* Customize behavior through user code.
I originally built this tool to generate my personal homepage, but I believe it can be of use to anyone who wants to
enjoy the freedom of building a static site from ground up, especially users of Metalsmith. Why craft metal when you can
be crafting gold?
## Usage ##
Goldsmith is at it's core, a pipeline-based file processor. Files are loaded from the source directory, processed by any
number of plugins, and are finally output to the destination directory. Rather than explaining the process in detail
conceptually, I will show some code samples which show how this tool can be used in practice.
* Start by copying files from a source directory to a destination directory (simplest possible use case):
```
goldsmith.Begin(srcDir).
End(dstDir)
```
* Now let's also convert our [Markdown](https://daringfireball.net/projects/markdown/) files to HTML using the
[markdown plugin](../goldsmith-plugins/markdown):
```
goldsmith.Begin(srcDir).
Chain(markdown.NewCommon()).
End(dstDir)
```
* If we are using *front matter* in our Markdown files, we can easily extract it by using the
[frontmatter plugin](../goldsmith-plugins/frontmatter):
```
goldsmith.Begin(srcDir).
Chain(frontmatter.New()).
Chain(markdown.NewCommon()).
End(dstDir)
```
* Next we want to run our generated HTML through a template to add a header, footer, and a menu; for this we
can use the [layout plugin](../goldsmith-plugins/layout):
```
goldsmith.Begin(srcDir).
Chain(frontmatter.New()).
Chain(markdown.NewCommon()).
Chain(layout.New(
layoutFiles, // array of paths for files containing template definitions
templateNameVar, // metadata variable that contains the name of the template to use
contentStoreVar, // metadata variable configured in template to insert content
defTemplateName, // name of a default template to use if one is not specified
userFuncs, // mapping of functions which can be executed from templates
)).
End(dstDir)
```
* Finally, let's [minify](https://en.wikipedia.org/wiki/Minification_(programming)) our files to reduce data transfer
and load times for our site's visitors using the [minify plugin](../goldsmith-plugins/minify).
```
goldsmith.Begin(srcDir).
Chain(frontmatter.New()).
Chain(markdown.NewCommon()).
Chain(layout.New(layoutFiles, templateNameVar, contentStoreVar, defTemplateName, userFuncs)).
Chain(minify.New()).
End(dstDir)
```
I hope this simple example effectively illustrates the conceptual simplicity of the Goldsmith pipeline-based processing
method. Files are injected into the stream at Goldsmith initialization, processed in parallel through a series of
plugins, and are finally written out to disk upon completion.
Files are guaranteed to flow through Goldsmith plugins in the same order, but not necessarily in the same sequence
relative to each other. Timing differences can cause certain files to finish ahead of others; fortunately this, along
with other threading characteristics of the tool is abstracted from the user. The execution, while appearing to be a
mere series chained methods, will process files using all of your system's cores.
## License ##
MIT

61
context.go
View File

@ -30,26 +30,17 @@ import (
type context struct { type context struct {
gs *goldsmith gs *goldsmith
plug Plugin
input, output chan *file input, output chan *file
} }
func newContext(gs *goldsmith) *context { func (ctx *context) chain() {
ctx := &context{gs: gs, output: make(chan *file)}
if len(gs.contexts) > 0 {
ctx.input = gs.contexts[len(gs.contexts)-1].output
}
gs.contexts = append(gs.contexts, ctx)
return ctx
}
func (ctx *context) chain(p Plugin) {
defer close(ctx.output) defer close(ctx.output)
init, _ := p.(Initializer) init, _ := ctx.plug.(Initializer)
accept, _ := p.(Accepter) accept, _ := ctx.plug.(Accepter)
proc, _ := p.(Processor) proc, _ := ctx.plug.(Processor)
fin, _ := p.(Finalizer) fin, _ := ctx.plug.(Finalizer)
if init != nil { if init != nil {
if err := init.Initialize(ctx); err != nil { if err := init.Initialize(ctx); err != nil {
@ -58,26 +49,28 @@ func (ctx *context) chain(p Plugin) {
} }
} }
var wg sync.WaitGroup if ctx.input != nil {
for i := 0; i < runtime.NumCPU(); i++ { var wg sync.WaitGroup
wg.Add(1) for i := 0; i < runtime.NumCPU(); i++ {
go func() { wg.Add(1)
defer wg.Done() go func() {
for f := range ctx.input { defer wg.Done()
if proc == nil || accept != nil && !accept.Accept(ctx, f) { for f := range ctx.input {
ctx.output <- f if proc == nil || accept != nil && !accept.Accept(ctx, f) {
} else { ctx.output <- f
if _, err := f.Seek(0, os.SEEK_SET); err != nil { } else {
ctx.gs.fault(f, err) if _, err := f.Seek(0, os.SEEK_SET); err != nil {
} ctx.gs.fault(f, err)
if err := proc.Process(ctx, f); err != nil { }
ctx.gs.fault(f, err) if err := proc.Process(ctx, f); err != nil {
ctx.gs.fault(f, err)
}
} }
} }
} }()
}() }
wg.Wait()
} }
wg.Wait()
if fin != nil { if fin != nil {
if err := fin.Finalize(ctx); err != nil { if err := fin.Finalize(ctx); err != nil {
@ -94,10 +87,6 @@ func (ctx *context) DispatchFile(f File) {
ctx.output <- f.(*file) ctx.output <- f.(*file)
} }
func (ctx *context) ReferenceFile(path string) {
ctx.gs.referenceFile(path)
}
func (ctx *context) SrcDir() string { func (ctx *context) SrcDir() string {
return ctx.gs.srcDir return ctx.gs.srcDir
} }

143
core.go Normal file
View File

@ -0,0 +1,143 @@
/*
* Copyright (c) 2015 Alex Yatskov <alex@foosoft.net>
* Author: Alex Yatskov <alex@foosoft.net>
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
* the Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
* IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package goldsmith
import (
"os"
"path/filepath"
"sync"
)
type goldsmith struct {
srcDir, dstDir string
contexts []*context
refs map[string]bool
errors []error
errorMtx sync.Mutex
}
func (gs *goldsmith) pushContext(plug Plugin) *context {
ctx := &context{
gs: gs,
plug: plug,
output: make(chan *file),
}
if len(gs.contexts) > 0 {
ctx.input = gs.contexts[len(gs.contexts)-1].output
}
gs.contexts = append(gs.contexts, ctx)
return ctx
}
func (gs *goldsmith) cleanupFiles() {
files := make(chan string)
dirs := make(chan string)
go scanDir(gs.dstDir, files, dirs)
for files != nil || dirs != nil {
var (
path string
ok bool
)
select {
case path, ok = <-files:
if !ok {
files = nil
continue
}
case path, ok = <-dirs:
if !ok {
dirs = nil
continue
}
default:
continue
}
relPath, _ := filepath.Rel(gs.dstDir, path)
if contained, _ := gs.refs[relPath]; contained {
continue
}
os.RemoveAll(path)
}
}
func (gs *goldsmith) exportFile(f *file) error {
if err := f.export(gs.dstDir); err != nil {
return err
}
pathSeg := cleanPath(f.path)
for {
gs.refs[pathSeg] = true
if pathSeg == "." {
break
}
pathSeg = filepath.Dir(pathSeg)
}
return nil
}
func (gs *goldsmith) fault(f *file, err error) {
gs.errorMtx.Lock()
defer gs.errorMtx.Unlock()
ferr := &Error{Err: err}
if f != nil {
ferr.Path = f.path
}
gs.errors = append(gs.errors, ferr)
}
//
// Goldsmith Implementation
//
func (gs *goldsmith) Chain(p Plugin) Goldsmith {
gs.pushContext(p)
return gs
}
func (gs *goldsmith) End(dstDir string) []error {
gs.dstDir = dstDir
for _, ctx := range gs.contexts {
go ctx.chain()
}
ctx := gs.contexts[len(gs.contexts)-1]
for f := range ctx.output {
gs.exportFile(f)
}
gs.cleanupFiles()
return gs.errors
}

30
file.go
View File

@ -28,17 +28,23 @@ import (
"io/ioutil" "io/ioutil"
"os" "os"
"path" "path"
"path/filepath"
) )
type file struct { type file struct {
path string path string
meta map[string]interface{} Meta map[string]interface{}
reader *bytes.Reader reader *bytes.Reader
asset string asset string
} }
func (f *file) export(dstPath string) error { func (f *file) export(dstDir string) error {
dstPath := filepath.Join(dstDir, f.path)
if len(f.asset) > 0 && fileCached(f.asset, dstPath) {
return nil
}
if err := os.MkdirAll(path.Dir(dstPath), 0755); err != nil { if err := os.MkdirAll(path.Dir(dstPath), 0755); err != nil {
return err return err
} }
@ -93,13 +99,23 @@ func (f *file) Path() string {
return f.path return f.path
} }
func (f *file) Meta() map[string]interface{} { func (f *file) Dir() string {
return f.meta return path.Dir(f.path)
} }
func (f *file) Apply(m map[string]interface{}) { func (f *file) Value(key string) (interface{}, bool) {
for key, value := range m { value, ok := f.Meta[key]
f.meta[key] = value return value, ok
}
func (f *file) SetValue(key string, value interface{}) {
f.Meta[key] = value
}
func (f *file) CopyValues(src File) {
rf := src.(*file)
for name, value := range rf.Meta {
f.SetValue(name, value)
} }
} }

185
goldsmith.go
View File

@ -23,129 +23,80 @@
package goldsmith package goldsmith
import ( import (
"os" "bytes"
"path/filepath" "io"
"sync"
) )
type goldsmith struct { type Goldsmith interface {
srcDir, dstDir string Chain(p Plugin) Goldsmith
contexts []*context End(dstDir string) []error
refs map[string]bool
refMtx sync.Mutex
errors []error
errorMtx sync.Mutex
} }
func (gs *goldsmith) queueFiles() { func Begin(srcDir string) Goldsmith {
files := make(chan string) gs := &goldsmith{srcDir: srcDir, refs: make(map[string]bool)}
go scanDir(gs.srcDir, files, nil) gs.Chain(new(loader))
ctx := newContext(gs)
go func() {
defer close(ctx.output)
for path := range files {
relPath, err := filepath.Rel(gs.srcDir, path)
if err != nil {
panic(err)
}
f := NewFileFromAsset(relPath, path)
ctx.DispatchFile(f)
}
}()
}
func (gs *goldsmith) cleanupFiles() {
files := make(chan string)
dirs := make(chan string)
go scanDir(gs.dstDir, files, dirs)
for files != nil || dirs != nil {
var (
path string
ok bool
)
select {
case path, ok = <-files:
if !ok {
files = nil
continue
}
case path, ok = <-dirs:
if !ok {
dirs = nil
continue
}
default:
continue
}
relPath, err := filepath.Rel(gs.dstDir, path)
if err != nil {
panic(err)
}
if contained, _ := gs.refs[relPath]; contained {
continue
}
os.RemoveAll(path)
}
}
func (gs *goldsmith) exportFile(f *file) error {
absPath := filepath.Join(gs.dstDir, f.path)
if err := f.export(absPath); err != nil {
return err
}
gs.referenceFile(f.path)
return nil
}
func (gs *goldsmith) referenceFile(path string) {
gs.refMtx.Lock()
defer gs.refMtx.Unlock()
path = cleanPath(path)
for {
gs.refs[path] = true
if path == "." {
break
}
path = filepath.Dir(path)
}
}
func (gs *goldsmith) fault(f *file, err error) {
gs.errorMtx.Lock()
gs.errors = append(gs.errors, &Error{f, err})
gs.errorMtx.Unlock()
}
//
// Goldsmith Implementation
//
func (gs *goldsmith) Chain(p Plugin) Goldsmith {
ctx := newContext(gs)
go ctx.chain(p)
return gs return gs
} }
func (gs *goldsmith) Complete() []error { type File interface {
ctx := gs.contexts[len(gs.contexts)-1] Path() string
for f := range ctx.output { Dir() string
gs.exportFile(f)
}
gs.cleanupFiles() Value(key string) (interface{}, bool)
return gs.errors SetValue(key string, value interface{})
CopyValues(src File)
Read(p []byte) (int, error)
WriteTo(w io.Writer) (int64, error)
Seek(offset int64, whence int) (int64, error)
} }
func NewFileFromData(path string, data []byte) File {
return &file{
path: path,
Meta: make(map[string]interface{}),
reader: bytes.NewReader(data),
}
}
func NewFileFromAsset(path, asset string) File {
return &file{
path: path,
Meta: make(map[string]interface{}),
asset: asset,
}
}
type Context interface {
DispatchFile(f File)
SrcDir() string
DstDir() string
}
type Error struct {
Err error
Path string
}
func (e Error) Error() string {
return e.Err.Error()
}
type Initializer interface {
Initialize(ctx Context) error
}
type Accepter interface {
Accept(ctx Context, f File) bool
}
type Processor interface {
Process(ctx Context, f File) error
}
type Finalizer interface {
Finalize(ctx Context) error
}
type Plugin interface{}

40
loader.go Normal file
View File

@ -0,0 +1,40 @@
/*
* Copyright (c) 2016 Alex Yatskov <alex@foosoft.net>
* Author: Alex Yatskov <alex@foosoft.net>
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
* the Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
* IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package goldsmith
import "path/filepath"
type loader struct{}
func (*loader) Initialize(ctx Context) error {
files := make(chan string)
go scanDir(ctx.SrcDir(), files, nil)
for path := range files {
relPath, _ := filepath.Rel(ctx.SrcDir(), path)
f := NewFileFromAsset(relPath, path)
ctx.DispatchFile(f)
}
return nil
}

106
types.go
View File

@ -1,106 +0,0 @@
/*
* Copyright (c) 2015 Alex Yatskov <alex@foosoft.net>
* Author: Alex Yatskov <alex@foosoft.net>
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to
* use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
* the Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
* IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package goldsmith
import (
"bytes"
"io"
)
type Goldsmith interface {
Chain(p Plugin) Goldsmith
Complete() []error
}
func New(srcDir, dstDir string) Goldsmith {
gs := &goldsmith{
srcDir: srcDir,
dstDir: dstDir,
refs: make(map[string]bool),
}
gs.queueFiles()
return gs
}
type File interface {
Path() string
Meta() map[string]interface{}
Apply(m map[string]interface{})
Read(p []byte) (int, error)
WriteTo(w io.Writer) (int64, error)
Seek(offset int64, whence int) (int64, error)
}
func NewFileFromData(path string, data []byte) File {
return &file{
path: path,
meta: make(map[string]interface{}),
reader: bytes.NewReader(data),
}
}
func NewFileFromAsset(path, asset string) File {
return &file{
path: path,
meta: make(map[string]interface{}),
asset: asset,
}
}
type Context interface {
DispatchFile(f File)
ReferenceFile(path string)
SrcDir() string
DstDir() string
}
type Error struct {
file File
err error
}
func (e *Error) Error() string {
return e.err.Error()
}
type Initializer interface {
Initialize(ctx Context) error
}
type Accepter interface {
Accept(ctx Context, file File) bool
}
type Finalizer interface {
Finalize(ctx Context) error
}
type Processor interface {
Process(ctx Context, f File) error
}
type Plugin interface{}

14
util.go
View File

@ -66,3 +66,17 @@ func scanDir(root string, files, dirs chan string) {
return nil return nil
}) })
} }
func fileCached(srcPath, dstPath string) bool {
srcStat, err := os.Stat(srcPath)
if err != nil {
return false
}
dstStat, err := os.Stat(dstPath)
if err != nil {
return false
}
return dstStat.ModTime().Unix() >= srcStat.ModTime().Unix()
}