Mark has answered the bit about getting the outer rectangle sorted once the inner stuff is in place (given the constraints, the fit and layer method seems the better fit). What I'm going to tackle is getting the inner boxes to position themselves with "line wrapping". This is not a complete solution, but if it seems along the right lines then I think that the rest would not be hard to do (but I want to check before doing that).
My comment to the question (with the quote from the real Italian Job) was along the right lines but just not quite far enough along. The positioning library can take quite complicated expressions - it is possible to do things like above={(\x > \y ? 3cm : -2cm)} of a.south - but it turns out that these are computed before the node is laid out. All the complicated positioning is done by the node code, not by the positioning code.
So we have to co-opt the node code to our nefarious purposes. Fortunately, this is relatively easy (if a little tedious to do in full). Nodes are positioned by specifying an anchor. That anchor is placed at (0,0) (the positioning library, or just TikZ, sets things up so that as far as the node is concerned, (0,0) is where the node is "at"). So if we want to place the node somewhere a little different, we simply define a rather complicated anchor. This entails either defining a new shape or using Martin Scharrer's code for adding new anchors to an existing shape. I've gone for the former and defined a "moveable rectangle". This rectangle has the property that its north west anchor can depend on a parameter. Well, actually a little computation. This computation can use the information about the node size, since it is computed while the node is being laid out. My computation is:
\pgfmathparse{(\mrwidth > 1cm ? "(a.north east)" : "(a.south west)")}
\let\mrvector=\pgfmathresult
This tests a macro called \mrwidth which happens to be set to the width of the node (mr = "moveable rectangle") (obviously, one could adapt it to make the height available as well). If the width is big, it returns (a.north east) and if small, (a.south west). The result is saved as \mrvector. This code is executed while the node is working out the north west anchor of the rectangle and the anchor is shifted according to the result. As node coordinates are "absolute", the result is that the north west anchor of the current node is placed at either the north east or the south west of the a node depending on its width. If the coordinates had been things like (0,2cm), this would have indicated a relative adjustment.
So the idea is to make a slightly more complicated test which looks to see if the width of the node is going to take it outside the given width and if so, anchor the node on the next line. This would need a little more work, but not overmuch, as there will typically be several nodes involved in the computation. But that's just bookkeeping.
I'll put the full code at the end of this answer. If it were all packaged up in a nice package, the syntax would be something like:
\begin{document}
\begin{tikzpicture}
\node[box] (a) {A};
\node[box,moveable rectangle,anchor=north west,moveable] at (a.south east) (b) {B};
\node[box,moveable rectangle,anchor=north west,moveable] at (a.south east) (c) {ABCDEFG};
\end{tikzpicture}
\end{document}
with result:

As you can see, the specifications to the two nodes are exactly the same, but one ends up alongside the a node and one just below it.
Here's the code so far. I've currently only modified the north west anchor. Obviously, the others will need doing as well. And stress testing and probably some assumptions I've made will be shown to be completely ludicrous ...
\documentclass{article}
%\url{http://tex.stackexchange.com/q/40234/86}
\usepackage{tikz}
\makeatletter
\tikzset{
box/.style={
minimum height=0.8cm,
draw,
rounded corners,
rectangle
},
moveable rectangle/condition/.code={%
\pgfmathparse{(\mrwidth > 1cm ? "(a.north east)" : "(a.south west)")}
\let\mrvector=\pgfmathresult
},
moveable/.is if=mr@moveable,
}
\def\mrvector{(0,0)}
\newif\ifmr@moveable
\pgfdeclareshape{moveable rectangle}{%
\inheritsavedanchors[from=rectangle]%
\anchor{north west}{%
\ifmr@moveable
\northeast
\pgf@xa=\pgf@x
\southwest
\advance\pgf@xa by -\pgf@x
\edef\mrwidth{\the\pgf@xa}%
\pgfkeys{/tikz/moveable rectangle/condition}%
\tikz@scan@one@point\pgfutil@firstofone\mrvector
\pgf@xa=-\pgf@x
\pgf@ya=-\pgf@y
\northeast
\advance\pgf@ya by \pgf@y
\southwest
\advance\pgf@xa by \pgf@x
\pgf@x=\pgf@xa
\pgf@y=\pgf@ya
\else
\northeast
\pgf@ya=\pgf@y
\southwest
\pgf@y=\pgf@ya
\fi
}
\inheritbackgroundpath[from=rectangle]
}
\makeatother
\begin{document}
\begin{tikzpicture}
\node[box] (a) {A};
\node[box,moveable rectangle,anchor=north west,moveable] at (a.south east) (b) {B};
\node[box,moveable rectangle,anchor=north west,moveable] at (a.south east) (c) {ABCDEFG};
\end{tikzpicture}
\end{document}
fitlibrary would require that you position your boxes manually first. However, it would be possible to code a macro or environment which takes the nodes and position them automatically. – Martin Scharrer Jan 06 '12 at 11:05\tikzstyleis deprecated. The official way is\tikzset{bigbox/.style={<style>},..}. – Martin Scharrer Jan 06 '12 at 11:06matrixnode style is for. The inner nodes are laid out in a grid (which can be customised) and then you have a "big" node outside (the matrix node) which you can also style. – Andrew Stacey Jan 06 '12 at 11:15matrixnode style (as well as with @MarkS.Everitt's answer) you need to know (and specify) the layout yourself. The way I read it is that the OP wants this to happen automatically. – Roelof Spijker Jan 06 '12 at 11:23fitis useful. – qubyte Jan 06 '12 at 11:46baseline=(current bounding box.center)int the options for the outer figure, i think. – Loic Rosnay Jan 06 '12 at 11:58[baseline=0pt]on the inner pictures. That overrides the[baseline=(outer.base)]inherited from the outer. – Andrew Stacey Jan 06 '12 at 12:20positioninglibrary that allows conditionals. – Andrew Stacey Jan 06 '12 at 14:27