Network-on-chip (NoC) has been proposed as a solution for the interconnection architecture design problem of System-on-chip (SoC) design in nanoscale technologies. NoC architecture for application specific SoC can be optimized by constructing custom topologies that are more suitable for the given application. In nanoscale technologies, the link energy consumption will constitute a considerable part of the total communication energy. Therefore, the total energy consumption of the NoC is strongly influenced by systemlevel floorplan. In this paper, we present a novel integer linear programming( ILP) based technique for joint optimization of NoC with system-level floorplan. We also present a clustering based heuristic technique to reduce the runtime of the ILP formulation. Experimental results with realistic benchmarks applications demonstrate superiority of custom NoC topologies generated by our techniques over mesh based networks, and high quality of the clustering based technique when compared with the ILP formulation.