High-dimensional quantum units of information, or qudits, can carry more than one quantum bit of information in a single degree of freedom, and can therefore be used to boost the performance of quantum communication and quantum computation protocols. A photon in a superposition of (2^N) time bins - a time-bin qudit - contains as much information as N qubits. Here, we show that N-qubit states encoded in a single time-bin qudit can be arbitrarily and deterministically generated, manipulated and measured using a number of linear optics elements that scales linearly with N, as opposed with prior proposals of single-qudit implementation of N-qubit logic, which typically requires (O(2^N)) elements. The simple and cost-effective implementation we propose can be used as a small-scale quantum processor. We then demonstrate a path towards scalability by interfacing distinct qudit processors to a matter qubit (atom or quantum dot spin) in an optical resonator. Such a cavity QED system allows for more advanced functionalities, such as single-qubit nondemolition measurement and two-qubit gates between distinct qudits. It could also enable quantum interfaces with other matter quantum nodes in the context of quantum networks and distributed quantum computing.