I mean something like this:
/ \
| x |
\ /
I looked up many related resources but failed.
Build your own... I should add that \roundrule.sty is not a CTAN package, but found at Is there such a thing as a `\mathrule`? (rounded endcaps). If one doesn't care about rounded endcaps, \rule can be substituted for \roundrule
EDITED by popular demand.
\documentclass{article}
\usepackage{roundrule}
\usepackage{graphicx}
\usepackage{stackengine}
\usepackage{scalerel}
\def\rlwd{.4pt}% \rule width
\def\lhexbrace{\kern1pt%
\setstackgap{S}{0pt}\def\stackalignment{l}
\ThisStyle{\scalerel*{%
\stackunder[-\rlwd]{%
\stackon[-\rlwd]{\roundrule{\rlwd}{4pt}}{\rotatebox{45}{\roundrule{4pt}{\rlwd}}}%
}{\rotatebox{-45}{\roundrule{4pt}{\rlwd}}}%
}{\SavedStyle[}}}
\def\rhexbrace{%
\setstackgap{S}{0pt}\def\stackalignment{r}
\ThisStyle{\scalerel*{%
\stackunder[-\rlwd]{%
\stackon[-\rlwd]{\roundrule{\rlwd}{4pt}}{\rotatebox{-45}{\roundrule{4pt}{\rlwd}}}%
}{\rotatebox{45}{\roundrule{4pt}{\rlwd}}}%
}{\SavedStyle[}}\kern1pt}
\begin{document}
$[ABC] \lhexbrace ABC\rhexbrace \lhexbrace xyz\rhexbrace
\scriptstyle\lhexbrace xyz\rhexbrace
\scriptscriptstyle\lhexbrace xyz\rhexbrace$\bigskip
\setstackEOL{\\}
~~\raisebox{-8pt}{/}\kern-1pt\fbox{\Shortstack[l]{The Force is strong\\ with this one.}}\\
$\lhexbrace{\kern-3pt-}\mathsf{o}{-\kern-3pt}\rhexbrace$
\end{document}

With just a little more work,
\setstackEOL{\\}
~~\raisebox{-8pt}{/}\kern-1pt\fbox{\Shortstack[l]{The Force is strong\\ with this one.}}\\
$\lhexbrace{\kern-3pt-}\stackinset{c}{}{c}{}{\fboxsep=0pt\fboxrule=0.1pt%
\fbox{\scalebox{.05}{\Burns}}}{$\mathsf{o}$}{-\kern-3pt}\rhexbrace$
simpsons.sty, and a lot of zoom, the truth can be revealed:

$\lhexbrace{\kern-3pt-}\mathsf{o}{-\kern-3pt}\rhexbrace$ will generate Darth Vader's "TIE advanced x1 fighter"
– Steven B. Segletes
May 05 '14 at 15:23
\rlwd can be adjusted to get the desired rule thickness. Thank you.
– Steven B. Segletes
May 06 '14 at 01:49
The symbols you are looking for are defined in unicode as (left side) U+27EC, U+2997, U+3014, and U+3018 and (right side) U+27ED, U+2998, U+3015, and U+3019.
If you are willing to switch to LuaLaTeX (or XeLaTeX, I guess), you could use the Asana Math Font. Just download the .otf file and insert it in C:\WINDOWS\Fonts.
% arara: lualatex
\documentclass{article}
\usepackage{unicode-math}
\setmathfont{Asana Math}
\begin{document}
black tortoise shell bracket
$\lblkbrbrak x\rblkbrbrak$
broken bracket
$\lbrbrak x\rbrbrak$
white tortoise shell bracket
$\Lbrbrak x\Rbrbrak$
\end{document}

The last version looks somehow broken to me. I will look, if I should report that somewhere.
Other possibilities are:


from XITS Math or:


from Cambria Math.
The vertical displacement comes from my screenshot. They will be well aligned when typeset.
The maintainer of Asana Math (Apostolos Syropoulos, who sends his kind regards and is happy for any feedback and issue report to asyropoulos at aol dot com) immediately fixed the white tortoise bracket. I installed the new opentype font and the xits opentype fonts and finally got the following 4 versions (Cambria Math needs to be purchased and therefore has not been tested by me):

% arara: lualatex
\documentclass{article}
\usepackage{unicode-math}
\begin{document}
\setmathfont{Asana-Math.otf}
$\lblkbrbrak x\rblkbrbrak$
$\lbrbrak x\rbrbrak$
$\Lbrbrak x\Rbrbrak$
\setmathfont{xits-Math.otf}
$\lblkbrbrak x\rblkbrbrak$
\end{document}
A TikZ answer (which works with pdflatex and does only need amsmath and tikz):
here is the code:
\documentclass[varwidth=true, border=2pt]{standalone}
\usepackage[ngerman]{babel}
\usepackage[utf8]{inputenc}
% Color
\usepackage{color}
\definecolor{lightgray}{gray}{0.95}
\definecolor{gray}{gray}{0.5}
% The left tortoise shell bracket
% Add the following code to your preamble
% <code for lbtsbrak>
\usepackage{amsmath}
\usepackage{tikz}
\newcommand\LBTSbrak{%
\mathopen{%
\mspace{1mu}%
\text{\lbtsbrak}%
\mspace{1mu}%
}%
}
\newcommand\lbtsbrak{%
\tikz[line cap=round,x=1ex,y=1ex,line width=0.3pt]
{\draw (0.4,2) -- (0.0,1.5) -- (0.0,0.5) -- (0.4,0.0);}%
}
% </code for lbtsbrak>
% Example how to use it
\begin{document}
{\textcolor{lightgray}{a}}$\LBTSbrak${\textcolor{lightgray}{b}}
\end{document}
pdflatex? – SaltyEgg May 05 '14 at 14:38