| 215 |
Teach the CLI to Talk Back |
The CLI is a daunting tool. It’s quick, powerful, but it’s also incredibly easy to screw things up in – either with a mistyped command, or a correctly typed command used at the wrong moment. This puts a lot of people off using it, but it doesn’t have to be this way.
If you’ve ever interacted with Slack’s Slackbot to set a reminder or ask a question, you’re basically using a command line interface, but it feels more like having a conversation. (My favourite Slack app is Lunch Train which helps with the thankless task of herding colleagues to a particular lunch venue on time.)
Same goes with voice-operated assistants like Alexa, Siri and Google Home. There are even games, like Lifeline, where you interact with a stranded astronaut via pseudo SMS, and KOMRAD where you chat with a Soviet AI.
I’m not aiming to build an AI here – my aspirations are a little more down to earth. What I’d like is to make the CLI a friendlier, more forgiving, and more intuitive tool for new or reluctant users. I want to teach it to talk back.
Interactive command lines in the wild
If you’ve used dev tools in the command line, you’ve probably already used an interactive prompt – something that asks you questions and responds based on your answers. Here are some examples:
Yeoman
If you have Yeoman globally installed, running yo will start a command prompt.
The prompt asks you what you’d like to do, and gives you options with how to proceed. Seasoned users will run specific commands for these options rather than go through this prompt, but it’s a nice way to start someone off with using the tool.
npm
If you’re a Node.js developer, you’re probably familiar with typing npm init to initialise a project. This brings up prompts that will populate a package.json manifest file for that project.
The alternative would be to expect the user to craft their own package.json, which is more error-prone since it’s in JSON format, so something as trivial as an extraneous comma can throw an error.
Snyk
Snyk is a dev tool that checks for known vulnerabilities in your dependencies. Running snyk wizard in the CLI brings up a list of all the known vulnerabilities, and gives you options on how to deal with it – such as patching the issue, applying a fix by upgrading the problematic dependency, or ignoring the issue (you are then prompted for a reason).
These decisions get mapped to the manifest and a .snyk file, and committed into the repo so that the settings are the same for everyone who uses that project.
I work at Snyk, and running the wizard is what made me think about building my own personal assistant in the command line to help me with some boring, repetitive tasks.
Writing your own
Something I do a lot is add bookmarks to styleguides.io – I pull down the entire repo, copy and paste a template YAML file, and edit to contents. Sometimes I get it wrong and break the site. So I’ve been putting together a tool to help me add bookmarks.
It’s called bookmarkbot – it’s a personal assistant squirrel called Mark who will collect and bury your bookmarks for safekeeping.*
*Fortunately, this metaphor also gives me a charming excuse for any situation where bookmarks sometimes get lost – it’s not my poorly-written code, honest, it’s just being realistic because sometimes squirrels forget where they buried things!
When you run bookmarkbot, it will ask you for some information, and save that information as a Markdown file in YAML format.
For this demo, I’m going to use a Node.js package called inquirer, which is a well supported tool for creating command line prompts. I like it because it has a bunch of different question types; from input, which asks for some text back, confirm which expects a yes/no response, or a list which gives you a set of options to choose from. You can even nest questions, Choose Your Own Adventure style.
Prerequisites
Node.js
npm
RubyGems (Only if you want to go as far as serving a static site for your bookmarks, and you want to use Jekyll for it)
Disclaimer
Bear in mind that this is a really simplified walkthrough. It doesn’t have any error states, and it doesn’t handle the situation where we save a file with the same name. But it gets you in a good place to start building out your tool.
Let’s go!
Create a new folder wherever you keep your projects, and give it an awesome name (I’ve called mine bookmarks and put it in the Sites directory because I’m unimaginative). Now cd to that directory.
cd Sites/bookmarks
Let’s use that example I gave earlier, the trusty npm init.
npm init
Pop in the information you’d like to provide, or hit ENTER to skip through and save the defaults. Your directory should now have a package.json file in it. Now let’s install some of the dependencies we’ll need.
npm install --save inquirer
npm install --save slugify
Next, add the following snippet to your package.json to tell it to run this file when you run npm start.
"scripts": {
…
"start": "node index.js"
}
That index.js file doesn’t exist yet, so let’s create it in the root of our folder, and add the following:
// Packages we need
var fs = require('fs'); // Creates our file (part of Node.js so doesn't need installing)
var inquirer = require('inquirer'); // The engine for our questions prompt
var slugify = require('slugify'); // Will turn a string into a usable filename
// The questions
var questions = [
{
type: 'input',
name: 'name',
message: 'What is your name?',
},
];
// The questions prompt
function askQuestions() {
// Ask questions
inquirer.prompt(questions).then(answers => {
// Things we'll need to generate the output
var name = answers.name;
// Finished asking questions, show the output
console.log('Hello ' + name + '!');
});
}
// Kick off the questions prompt
askQuestions();
This is just some barebones where we’re including the inquirer package we installed earlier. I’ve stored the questions in a variable, and the askQuestions function will prompt the user for their name, and then print “Hello <your name>” in the console.
Enough setup, let’s see some magic. Save the file, go back to the command line and run npm start.
Extending what we’ve learnt
At the moment, we’re just saving a name to a file, which isn’t really achieving our goal of saving bookmarks. We don’t want our tool to forget our information every time we talk to it – we need to save it somewhere. So I’m going to add a little function to write the output to a file.
Saving to a file
Create a folder in your project’s directory called _bookmarks. This is where the bookmarks will be saved.
I’ve replaced my questions array, and instead of asking for a name, I’ve extended out the questions, asking to be provided with a link and title (as a regular input type), a list of tags (using inquirer’s checkbox type), and finally a description, again, using the input type.
So this is how my code looks now:
// Packages we need
var fs = require('fs'); // Creates our file
var inquirer = require('inquirer'); // The engine for our questions prompt
var slugify = require('slugify'); // Will turn a string into a usable filename
// The questions
var questions = [
{
type: 'input',
name: 'link',
message: 'What is the url?',
},
{
type: 'input',
name: 'title',
message: 'What is the title?',
},
{
type: 'checkbox',
name: 'tags',
message: 'Would you like me to add any tags?',
choices: [
{ name: 'frontend' },
{ name: 'backend' },
{ name: 'security' },
{ name: 'design' },
{ name: 'process' },
{ name: 'business' },
],
},
{
type: 'input',
name: 'description',
message: 'How about a description?',
},
];
// The questions prompt
function askQuestions() {
// Say hello
console.log('🐿 Oh, hello! Found something you want me to bookmark?\n');
// Ask questions
inquirer.prompt(questions).then((answers) => {
// Things we'll need to generate the output
var title = answers.title;
var link = answers.link;
var tags = answers.tags + '';
var description = answers.description;
var output = '---\n' +
'title: "' + title + '"\n' +
'link: "' + link + '"\n' +
'tags: [' + tags + ']\n' +
'---\n' + description + '\n';
// Finished asking questions, show the output
console.log('\n🐿 All done! Here is what I\'ve written down:\n');
console.log(output);
// Things we'll need to generate the filename
var slug = slugify(title);
var filename = '_bookmarks/' + slug + '.md';
// Write the file
fs.writeFile(filename, output, function () {
console.log('\n🐿 Great! I have saved your bookmark to ' + filename);
});
});
}
// Kick off the questions prompt
askQuestions();
The output is formatted into YAML metadata as a Markdown file, which will allow us to turn it into a static HTML file using a build tool later. Run npm start again and have a look at the file it outputs.
Getting confirmation
Before the user makes critical changes, it’s good to verify those changes first. We’re going to add a confirmation step to our tool, before writing the file. More seasoned CLI users may favour speed over a “hey, can you wait a sec and just check this is all ok” step, but I always think it’s worth adding one so you can occasionally save someone’s butt.
So, underneath our questions array, let’s add a confirmation array.
// Packages we need
…
// The questions
…
// Confirmation questions
var confirm = [
{
type: 'confirm',
name: 'confirm',
message: 'Does this look good?',
},
];
// The questions prompt
…
As we’re adding the confirm step before the file gets written, we’ll need to add the following inside the askQuestions function:
// The questions prompt
function askQuestions() {
// Say hello
…
// Ask questions
inquirer.prompt(questions).then((answers) => {
…
// Things we'll need to generate the output
…
// Finished asking questions, show the output
…
// Confirm output is correct
inquirer.prompt(confirm).then(answers => {
// Things we'll need to generate the filename
var slug = slugify(title);
var filename = '_bookmarks/' + slug + '.md';
if (answers.confirm) {
// Save output into file
fs.writeFile(filename, output, function () {
console.log('\n🐿 Great! I have saved your bookmark to ' +
filename);
});
} else {
// Ask the questions again
console.log('\n🐿 Oops, let\'s try again!\n');
askQuestions();
}
});
});
}
// Kick off the questions prompt
askQuestions();
Now run npm start and give it a go!
Typing y will write the file, and n will take you back to the start. Ideally, I’d store the answers already given as defaults so the user doesn’t have to start from scratch, but I want to keep this demo simple.
Serving the files
Now that your bookmarking tool is successfully saving formatted Markdown files to a folder, the next step is to serve those files in a way that lets you share them online. The easiest way to do this is to use a static-site generator to convert your YAML files into HTML, and pop them all on one page. Now, you’ve got a few options here and I don’t want to force you down any particular path, as there are plenty out there – it’s just a case of using the one you’re most comfortable with.
I personally favour Jekyll because of its tight integration with GitHub Pages – I don’t want to mess around with hosting and deployment, so it’s really handy to have my bookmarks publish themselves on my site as soon as I commit and push them using Git.
I’ll give you a very brief run-through of how I’m doing this with bookmarkbot, but I recommend you read my Get Started With GitHub Pages (Plus Bonus Jekyll) guide if you’re unfamiliar with them, because I’ll be glossing over some bits that are already covered in there.
Setting up a build tool
If you haven’t already, install Jekyll and Bundler globally through RubyGems. Jekyll is our static-site generator, and Bundler is what we use to install Ruby dependencies.
gem install jekyll bundler
In my project folder, I’m going to run the following which will install the Jekyll files we’ll need to build our listing page. I’m using --force, otherwise it will complain that the directory isn’t empty.
jekyll new . --force
If you check your project folder, you’ll see a bunch of new files. Now run the following to start the server:
bundle exec jekyll serve
This will build a new directory called _site. This is where your static HTML files have been generated. Don’t touch anything in this folder because it will get overwritten the next time you build.
Now that serve is running, go to http://127.0.0.1:4000/ and you’ll see the default Jekyll page and know that things are set up right. Now, instead, we want to see our list of bookmarks that are saved in the _bookmarks directory (make sure you’ve got a few saved). So let’s get that set up next.
Open up the _config.yml file that Jekyll added earlier. In here, we’re going to tell it about our bookmarks. Replace everything in your _config.yml file with the following:
title: My Bookmarks
description: These are some of my favourite articles about the web.
markdown: kramdown
baseurl: /bookmarks # This needs to be the same name as whatever you call your repo on GitHub.
collections:
- bookmarks
This will make Jekyll aware of our _bookmarks folder so that we can call it later. Next, create a new directory and file at _layouts/home.html and paste in the following.
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>{{site.title}}</title>
<meta name="description" content="{{site.description}}">
</head>
<body>
<h1>{{site.title}}</h1>
<p>{{site.description}}</p>
<ul>
{% for bookmark in site.bookmarks %}
<li>
<a href="{{bookmark.link}}">
<h2>{{bookmark.title}}</h2>
</a>
{{bookmark.content}}
{% if bookmark.tags %}
<ul>
{% for tags in bookmark.tags %}<li>{{tags}}</li>{% endfor %}
</ul>
{% endif %}
</li>
{% endfor %}
</ul>
</body>
</html>
Restart Jekyll for your config changes to kick in, and go to the url it provides you (probably http://127.0.0.1:4000/bookmarks, unless you gave something different as your baseurl).
It’s a decent start – there’s a lot more we can do in this area but now we’ve got a nice list of all our bookmarks, let’s get it online!
If you want to use GitHub Pages to host your files, your first step is to push your project to GitHub. Go to your repository and click “settings”. Scroll down to the section labelled “GitHub Pages”, and from here you can enable it. Select your master branch, and it will provide you with a url to view your published pages.
What next?
Now that you’ve got a framework in place for publishing bookmarks, you can really go to town on your listing page and make it your own. First thing you’ll probably want to do is add some CSS, then when you’ve added a bunch of bookmarks, you’ll probably want to have some filtering in place for the tags, perhaps extend the types of questions that you ask to include an image (if you’re feeling extra-fancy, you could just ask for a url and pull in metadata from the site itself). Maybe you’ve got an idea that doesn’t involve bookmarks at all.
You could use what you’ve learnt to build a place where you can share quotes, a list of your favourite restaurants, or even Christmas gift ideas.
Here’s one I made earlier
My demo, bookmarkbot, is on GitHub, and I’ve reused a lot of the code from styleguides.io. Feel free to grab bits of code from there, and do share what you end up making! |
2017 |
Anna Debenham |
annadebenham |
2017-12-11T00:00:00+00:00 |
https://24ways.org/2017/teach-the-cli-to-talk-back/ |
code |
| 244 |
It’s Beginning to Look a Lot Like XSSmas |
I dread the office Secret Santa. I have a knack for choosing well-meaning but inappropriate presents, like a bottle of port for a teetotaller, a cheese-tasting experience for a vegan, or heaven forbid, Spurs socks for an Arsenal supporter. Ok, the last one was intentional.
It’s the same with gifting code. Once, I made a pattern library for A List Apart which I open sourced, and a few weeks later, a glaring security vulnerability was found in it. My gift was so generous that it enabled unrestricted access to any file on any public-facing server that hosted it.
With platforms like GitHub and npm, giving the gift of code is so easy it’s practically a no-brainer. This giant, open source yankee swap helps us do our jobs without starting from scratch with every project. But like any gift-giving, it’s also risky.
Vulnerabilities and Open Source
Open source code is not inherently more or less vulnerable than closed-source code. What makes it higher risk is that the same piece of code gets reused in lots of places, meaning a hacker can use the same exploit mechanism on the same vulnerable code in different apps.
Graph showing the number of open source vulnerabilities published per year, from the State of Open Source Security 2017
In the first 24 ways article this year, Katie referenced a few different types of vulnerability:
Cross-site Request Forgery (also known as CSRF)
SQL Injection
Cross-site Scripting (also known as XSS)
There are many more types of vulnerability, and those that live under the same category share similarities.
For example, my favourite – is it weird to have a favourite vulnerability? – is Cross Site Scripting (XSS), which allows for the injection of scripts into web pages. This is a really common vulnerability often unwittingly added by developers. OWASP (the Open Web Application Security Project) wrote a great article about how to prevent opening the door to XSS attacks – share it generously with your colleagues.
Most vulnerabilities like this are not added intentionally – they’re doors left ajar due to the way something has been scripted, like the over-generous code in my pattern library.
Others, though, are added intentionally. A few months ago, a hacker, disguised as a helpful elf, offered to take over the maintenance of a popular npm package that had been unmaintained for a couple of years. The owner had moved onto other projects, and was keen to see it continue to be maintained by someone else, so transferred ownership. Fast-forward 3 months, it was discovered that the individual had quietly added a malicious package to the codebase, and the obfuscated code in it had been unwittingly installed onto thousands of apps. The code added was designed to harvest Bitcoin if it was run alongside another application. It was only spotted due to a developer’s curiosity.
Another tactic to get developers to unwittingly install malicious packages into their codebase is “typosquatting” – back in August last year, npm reported that a user had been publishing packages with very similar names to popular packages (for example, crossenv instead of cross-env).
This is a big wakeup call for open source maintainers. Techniques like this are likely to be used more as the maintenance of open source libraries becomes an increasing burden to their owners. After all, starting a new project often has a greater reward than maintaining an existing one, but remember, an open source library is for life, not just for Christmas.
Santa’s on his sleigh
If you use open source libraries, chances are that these libraries also use open source libraries. Your app may only have a handful of dependencies, but tucked in the back of that sleigh may be a whole extra sack of dependencies known as deep dependencies (ones that you didn’t directly install, but are dependencies of that dependency), and these can contain vulnerabilities too.
Let’s look at the npm package santa as an example. santa has 8 direct dependencies listed on npm. That seems pretty manageable. But that’s just the tip of the iceberg – have a look at the full dependency tree which contains 109 dependencies – more dependencies than there are Christmas puns in this article. Only one of these direct dependencies has a vulnerability (at the time of writing), but there are actually 13 other known vulnerabilities in santa, which have been introduced through its deeper dependencies.
Fixing vulnerabilities – the ultimate christmas gift
If you’re a maintainer of open source libraries, taking good care of them is the ultimate gift you can give. Keep your dependencies up to date, use a security tool to monitor and alert you when new vulnerabilities are found in your code, and fix or patch them promptly. This will help keep the whole open source ecosystem healthy.
When you find out about a new vulnerability, you have some options:
Fix the vulnerability via an upgrade
You can often fix a vulnerability by upgrading the library to the latest version. Make sure you’re using software that monitors your dependencies for new security issues and lets you know when a fix is ready, otherwise you may be unwittingly using a vulnerable version.
Patch the vulnerable code
Sometimes, a fix for a vulnerable library isn’t possible. This is often the case when a library is no longer being maintained, or the version of the library being used might be so out of date that upgrading it would cause a breaking change. Patches are bits of code that will fix that particular issue, but won’t change anything else.
Switch to a different library
If the library you’re using has no fix or patch, you may be better of switching it out for another one, particularly if it looks like it’s being unmaintained.
Responsibly disclosing vulnerabilities
Knowing how to responsibly disclose vulnerabilities is something I’m ashamed to admit that I didn’t know about before I joined a security company. But it’s so important! On discovering a new vulnerability, a developer has a few options:
A malicious developer will exploit that vulnerability for their own gain.
A reckless (or inexperienced) developer will disclose that vulnerability to the world without following a responsible disclosure process. This opens the door to an unethical developer exploiting the vulnerability. At Snyk, we monitor social media for mentions of newly found vulnerabilities so we can add them to our database and share fixes before they get exploited.
An ethical and aware developer will follow what’s known as a “responsible disclosure process”. They will contact the maintainer of the code privately, allowing reasonable time for them to release a fix for the issue and to give others who use that vulnerable code a chance to fix it too.
It’s important to understand this process if you’re a maintainer or contributor of code. It can be daunting when a report comes in, but understanding and following the right steps will help reduce the risk to the people who use that code.
So what does responsible disclosure look like? I’ll take Node.js’s security disclosure policy as an example. They ask that all security issues that are found in Node.js are reported there. (There’s a separate process for bug found in third-party npm packages). Once you’ve reported a vulnerability, they promise to acknowledge it within 24 hours, and to give a more detailed response within 48 hours. If they find that the issue is indeed a security bug, they’ll give you regular updates about the progress they’re making towards fixing it. As part of this, they’ll figure out which versions are affected, and prepare fixes for them. They’ll assign the vulnerability a CVE (Common Vulnerabilities and Exposures) ID and decide on an embargo date for public disclosure. On the date of the embargo, they announce the vulnerability in their Node.js security mailing list and deploy fixes to nodejs.org.
Tim Kadlec published an in-depth article about responsible disclosures if you’re interested in knowing more. It has some interesting horror stories of what happened when the disclosure process was not followed.
Encourage responsible disclosure
Add a SECURITY.md file to your project so someone who wants to message you about a vulnerability can do so without having to hunt around for contact details. Last year, Snyk published a State of Open Source Security report that found 79.5% of maintainers do not have a public disclosure policy. Those that did were considerably more likely to get notified privately about a vulnerability – 73% of maintainers who had one had been notified, vs 21% of maintainers who hadn’t published one one.
Stats from the State of Open Source Security 2017
Bug bounties
Some companies run bug bounties to encourage the responsible disclosure of vulnerabilities. By offering a reward for finding and safely disclosing a vulnerability, it also reduces the enticement of exploiting a vulnerability over reporting it and getting a quick cash reward. Hackerone is a community of ethical hackers who pentest apps that have signed up for the scheme and get paid when they find a new vulnerability. Wordpress is one such participant, and you can see the long list of vulnerabilities that have been disclosed as part of that program.
If you don’t have such a bounty, be prepared to get the odd vulnerability extortion email. Scott Helme, who founded securityheaders.com and report-uri.com, wrote a post about some of the requests he gets for a report about a critical vulnerability in exchange for money.
On one hand, I want to be as responsible as possible and if my users are at risk then I need to know and patch this issue to protect them. On the other hand this is such irresponsible and unethical behaviour that interacting with this person seems out of the question.
A gift worth giving
It’s time to brush the dust off those old gifts that we shared and forgot about. Practice good hygiene and run them through your favourite security tool – I’m just a little biased towards Snyk, but as Katie mentioned, there’s also npm audit if you use Node.js, and most source code managers like GitHub and GitLab have basic vulnerability alert capabilities.
Stats from the State of Open Source Security 2017
Most importantly, patch or upgrade away those vulnerabilities away, and if you want to share that Christmas spirit, open fixes for your favourite open source projects, too. |
2018 |
Anna Debenham |
annadebenham |
2018-12-17T00:00:00+00:00 |
https://24ways.org/2018/its-beginning-to-look-a-lot-like-xssmas/ |
code |